[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4081 - Unstable!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4081 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4081/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudDeleteByQuery

Error Message:
Could not find collection:test_col

Stack Trace:
java.lang.AssertionError: Could not find collection:test_col
        at __randomizedtesting.SeedInfo.seed([4370360EC14CB8B1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
        at org.apache.solr.cloud.TestCloudDeleteByQuery.createMiniSolrCloudCluster(TestCloudDeleteByQuery.java:117)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudDeleteByQuery

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([4370360EC14CB8B1]:0)
        at org.apache.solr.cloud.TestCloudDeleteByQuery.afterClass(TestCloudDeleteByQuery.java:88)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11940 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestCloudDeleteByQuery
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/init-core-data-001
   [junit4]   2> 1154246 WARN  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=32 numCloses=32
   [junit4]   2> 1154246 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 1154250 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1154251 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001
   [junit4]   2> 1154251 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1154252 INFO  (Thread-3065) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1154252 INFO  (Thread-3065) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1154255 ERROR (Thread-3065) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1154354 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:52855
   [junit4]   2> 1154398 INFO  (jetty-launcher-2731-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1154398 INFO  (jetty-launcher-2731-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1154398 INFO  (jetty-launcher-2731-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1154398 INFO  (jetty-launcher-2731-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1154399 INFO  (jetty-launcher-2731-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1154418 INFO  (jetty-launcher-2731-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2352a658{/solr,null,AVAILABLE}
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31447002{/solr,null,AVAILABLE}
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c9030b0{/solr,null,AVAILABLE}
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@cba96cf{HTTP/1.1,[http/1.1]}{127.0.0.1:52858}
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-5) [    ] o.e.j.s.Server Started @1157825ms
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52858}
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1e8b41e3{HTTP/1.1,[http/1.1]}{127.0.0.1:52857}
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-1) [    ] o.e.j.s.Server Started @1157825ms
   [junit4]   2> 1154419 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52857}
   [junit4]   2> 1154419 ERROR (jetty-launcher-2731-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1154420 ERROR (jetty-launcher-2731-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3271b6f8{HTTP/1.1,[http/1.1]}{127.0.0.1:52859}
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-4) [    ] o.e.j.s.Server Started @1157825ms
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52859}
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1154420 ERROR (jetty-launcher-2731-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T09:32:57.587Z
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1154420 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T09:32:57.587Z
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@52734ed9{/solr,null,AVAILABLE}
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T09:32:57.588Z
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@536b800{HTTP/1.1,[http/1.1]}{127.0.0.1:52860}
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-3) [    ] o.e.j.s.Server Started @1157827ms
   [junit4]   2> 1154421 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52860}
   [junit4]   2> 1154421 ERROR (jetty-launcher-2731-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1154422 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1154422 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1154422 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1154422 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T09:32:57.589Z
   [junit4]   2> 1154423 INFO  (jetty-launcher-2731-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@68ad6c58{/solr,null,AVAILABLE}
   [junit4]   2> 1154427 INFO  (jetty-launcher-2731-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@47eb3298{HTTP/1.1,[http/1.1]}{127.0.0.1:52862}
   [junit4]   2> 1154427 INFO  (jetty-launcher-2731-thread-2) [    ] o.e.j.s.Server Started @1157833ms
   [junit4]   2> 1154428 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52862}
   [junit4]   2> 1154428 ERROR (jetty-launcher-2731-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1154429 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1154429 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1154429 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1154429 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T09:32:57.596Z
   [junit4]   2> 1154430 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1154430 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1154430 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1154438 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1154440 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb565f6cd0003, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1154441 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb565f6cd0002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1154443 INFO  (jetty-launcher-2731-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52855/solr
   [junit4]   2> 1154443 INFO  (jetty-launcher-2731-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52855/solr
   [junit4]   2> 1154443 INFO  (jetty-launcher-2731-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52855/solr
   [junit4]   2> 1154445 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1154449 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb565f6cd0004, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1154452 INFO  (jetty-launcher-2731-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52855/solr
   [junit4]   2> 1154452 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb565f6cd0006, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1154455 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb565f6cd0005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1154457 INFO  (jetty-launcher-2731-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52855/solr
   [junit4]   2> 1154522 INFO  (jetty-launcher-2731-thread-1) [n:127.0.0.1:52857_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1154522 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1154522 INFO  (jetty-launcher-2731-thread-2) [n:127.0.0.1:52862_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1154522 INFO  (jetty-launcher-2731-thread-5) [n:127.0.0.1:52858_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1154523 INFO  (jetty-launcher-2731-thread-3) [n:127.0.0.1:52860_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1154523 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52859_solr
   [junit4]   2> 1154525 INFO  (jetty-launcher-2731-thread-1) [n:127.0.0.1:52857_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52857_solr
   [junit4]   2> 1154525 INFO  (jetty-launcher-2731-thread-5) [n:127.0.0.1:52858_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52858_solr
   [junit4]   2> 1154525 INFO  (jetty-launcher-2731-thread-2) [n:127.0.0.1:52862_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52862_solr
   [junit4]   2> 1154525 INFO  (jetty-launcher-2731-thread-3) [n:127.0.0.1:52860_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52860_solr
   [junit4]   2> 1154526 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.c.Overseer Overseer (id=98152741432262667-127.0.0.1:52859_solr-n_0000000000) starting
   [junit4]   2> 1154528 INFO  (zkCallback-2752-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1154528 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1154528 INFO  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1154528 INFO  (zkCallback-2758-thread-1-processing-n:127.0.0.1:52862_solr) [n:127.0.0.1:52862_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1154528 INFO  (zkCallback-2751-thread-1-processing-n:127.0.0.1:52860_solr) [n:127.0.0.1:52860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1154567 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52859_solr
   [junit4]   2> 1154579 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1154579 INFO  (zkCallback-2758-thread-1-processing-n:127.0.0.1:52862_solr) [n:127.0.0.1:52862_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1154579 INFO  (zkCallback-2752-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1154579 INFO  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1154580 INFO  (zkCallback-2751-thread-1-processing-n:127.0.0.1:52860_solr) [n:127.0.0.1:52860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1154678 INFO  (jetty-launcher-2731-thread-5) [n:127.0.0.1:52858_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52858.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154683 INFO  (jetty-launcher-2731-thread-3) [n:127.0.0.1:52860_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52860.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154692 INFO  (jetty-launcher-2731-thread-5) [n:127.0.0.1:52858_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52858.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154692 INFO  (jetty-launcher-2731-thread-5) [n:127.0.0.1:52858_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52858.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154693 INFO  (jetty-launcher-2731-thread-1) [n:127.0.0.1:52857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52857.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154691 INFO  (jetty-launcher-2731-thread-2) [n:127.0.0.1:52862_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52862.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154693 INFO  (jetty-launcher-2731-thread-5) [n:127.0.0.1:52858_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/.
   [junit4]   2> 1154706 INFO  (jetty-launcher-2731-thread-3) [n:127.0.0.1:52860_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52860.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154706 INFO  (jetty-launcher-2731-thread-3) [n:127.0.0.1:52860_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52860.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154708 INFO  (jetty-launcher-2731-thread-1) [n:127.0.0.1:52857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52857.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154708 INFO  (jetty-launcher-2731-thread-3) [n:127.0.0.1:52860_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/.
   [junit4]   2> 1154709 INFO  (jetty-launcher-2731-thread-1) [n:127.0.0.1:52857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52857.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154709 INFO  (jetty-launcher-2731-thread-1) [n:127.0.0.1:52857_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/.
   [junit4]   2> 1154710 INFO  (jetty-launcher-2731-thread-2) [n:127.0.0.1:52862_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52862.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154710 INFO  (jetty-launcher-2731-thread-2) [n:127.0.0.1:52862_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52862.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154711 INFO  (jetty-launcher-2731-thread-2) [n:127.0.0.1:52862_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node2/.
   [junit4]   2> 1154796 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52859.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154800 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52859.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154800 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52859.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1154802 INFO  (jetty-launcher-2731-thread-4) [n:127.0.0.1:52859_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node4/.
   [junit4]   2> 1154919 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 1154920 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52855/solr ready
   [junit4]   2> 1155857 INFO  (qtp1993358908-14191) [n:127.0.0.1:52857_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&collection.configName=solrCloudCollectionConfig&name=test_col&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1155862 INFO  (OverseerThreadFactory-5962-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 1155998 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 1156000 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1156002 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 1156003 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1156003 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin
   [junit4]   2> 1156004 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1156005 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard2_replica_n1&action=CREATE&numShards=2&shard=shard2&wt=javabin
   [junit4]   2> 1156005 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1156136 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1156136 INFO  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1156136 INFO  (zkCallback-2758-thread-1-processing-n:127.0.0.1:52862_solr) [n:127.0.0.1:52862_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1156137 INFO  (zkCallback-2751-thread-1-processing-n:127.0.0.1:52860_solr) [n:127.0.0.1:52860_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1157047 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1157048 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1157050 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1157102 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1157110 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.s.IndexSchema [test_col_shard1_replica_n2] Schema name=test
   [junit4]   2> 1157117 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.s.IndexSchema [test_col_shard2_replica_n2] Schema name=test
   [junit4]   2> 1157170 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.s.IndexSchema [test_col_shard2_replica_n1] Schema name=test
   [junit4]   2> 1157232 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema [test_col_shard1_replica_n1] Schema name=test
   [junit4]   2> 1157373 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1157390 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1157399 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1157418 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica_n2' using configuration from collection test_col, trusted=true
   [junit4]   2> 1157419 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52857.solr.core.test_col.shard2.replica_n2' (registry 'solr.core.test_col.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1157419 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1157419 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.SolrCore [[test_col_shard2_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/test_col_shard2_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/./test_col_shard2_replica_n2/data/]
   [junit4]   2> 1157422 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7746923792095777]
   [junit4]   2> 1157430 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica_n2' using configuration from collection test_col, trusted=true
   [junit4]   2> 1157431 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52858.solr.core.test_col.shard1.replica_n2' (registry 'solr.core.test_col.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1157431 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1157431 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SolrCore [[test_col_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/test_col_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/data/]
   [junit4]   2> 1157434 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7746923792095777]
   [junit4]   2> 1157459 WARN  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1157460 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica_n1' using configuration from collection test_col, trusted=true
   [junit4]   2> 1157461 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52860.solr.core.test_col.shard2.replica_n1' (registry 'solr.core.test_col.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1157461 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1157461 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.SolrCore [[test_col_shard2_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/test_col_shard2_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/data/]
   [junit4]   2> 1157465 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7746923792095777]
   [junit4]   2> 1157488 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1157492 WARN  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1157533 WARN  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1157590 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica_n1' using configuration from collection test_col, trusted=true
   [junit4]   2> 1157591 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52862.solr.core.test_col.shard1.replica_n1' (registry 'solr.core.test_col.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66697963
   [junit4]   2> 1157591 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1157591 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.c.SolrCore [[test_col_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node2/test_col_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node2/./test_col_shard1_replica_n1/data/]
   [junit4]   2> 1157598 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7746923792095777]
   [junit4]   2> 1157606 WARN  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1157641 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1157641 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1157643 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1157643 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1157644 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1157644 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1157650 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1157644 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1157656 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1157656 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1157656 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1157656 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1157662 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3471964455226273]
   [junit4]   2> 1157663 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3471964455226273]
   [junit4]   2> 1157663 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3471964455226273]
   [junit4]   2> 1157663 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@10dd9afd[test_col_shard1_replica_n2] main]
   [junit4]   2> 1157664 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3312a025[test_col_shard2_replica_n1] main]
   [junit4]   2> 1157665 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 1157666 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 1157666 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 1157667 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 1157667 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1157667 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570443866494992384
   [junit4]   2> 1157667 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1157669 INFO  (searcherExecutor-5974-thread-1-processing-n:127.0.0.1:52858_solr x:test_col_shard1_replica_n2 s:shard1 c:test_col) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SolrCore [test_col_shard1_replica_n2] Registered new searcher Searcher@10dd9afd[test_col_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1157672 INFO  (searcherExecutor-5975-thread-1-processing-n:127.0.0.1:52860_solr x:test_col_shard2_replica_n1 s:shard2 c:test_col) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.SolrCore [test_col_shard2_replica_n1] Registered new searcher Searcher@3312a025[test_col_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1157673 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570443866501283840
   [junit4]   2> 1157674 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@741c74e7[test_col_shard2_replica_n2] main]
   [junit4]   2> 1157676 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 1157677 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 1157683 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9994ms
   [junit4]   2> 1157684 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1157686 INFO  (searcherExecutor-5973-thread-1-processing-n:127.0.0.1:52857_solr x:test_col_shard2_replica_n2 s:shard2 c:test_col) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.SolrCore [test_col_shard2_replica_n2] Registered new searcher Searcher@741c74e7[test_col_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1157686 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570443866514915328
   [junit4]   2> 1157694 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1157694 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1157695 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52860/solr/test_col_shard2_replica_n1/
   [junit4]   2> 1157696 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.PeerSync PeerSync: core=test_col_shard2_replica_n1 url=http://127.0.0.1:52860/solr START replicas=[http://127.0.0.1:52857/solr/test_col_shard2_replica_n2/] nUpdates=100
   [junit4]   2> 1157697 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.u.PeerSync PeerSync: core=test_col_shard2_replica_n1 url=http://127.0.0.1:52860/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1157699 INFO  (qtp1993358908-14194) [n:127.0.0.1:52857_solr c:test_col s:shard2 r:core_node1 x:test_col_shard2_replica_n2] o.a.s.c.S.Request [test_col_shard2_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1157700 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1157700 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1157700 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1157708 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52860/solr/test_col_shard2_replica_n1/ shard2
   [junit4]   2> 1157731 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.c.SolrCore [test_col_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3791a234
   [junit4]   2> 1157731 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.test_col.shard1.replica_n1, tag=932291124
   [junit4]   2> 1157734 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.test_col.shard1.leader, tag=932291124
   [junit4]   2> 1157741 ERROR (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'test_col_shard1_replica_n1': Unable to create core [test_col_shard1_replica_n1] Caused by: invalid API spec: apispec/emptySpec.json
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:948)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:747)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unable to create core [test_col_shard1_replica_n1]
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:994)
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:914)
   [junit4]   2> ... 34 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: invalid API spec: apispec/emptySpec.json
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1002)
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:855)
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:979)
   [junit4]   2> ... 35 more
   [junit4]   2> Caused by: java.lang.RuntimeException: invalid API spec: apispec/emptySpec.json
   [junit4]   2> at org.apache.solr.common.util.ValidatingJsonMap.parse(ValidatingJsonMap.java:318)
   [junit4]   2> at org.apache.solr.common.util.Utils.lambda$getSpec$0(Utils.java:319)
   [junit4]   2> at org.apache.solr.api.Api.getSpec(Api.java:65)
   [junit4]   2> at org.apache.solr.api.ApiBag.register(ApiBag.java:73)
   [junit4]   2> at org.apache.solr.core.PluginBag.put(PluginBag.java:217)
   [junit4]   2> at org.apache.solr.core.PluginBag.init(PluginBag.java:274)
   [junit4]   2> at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)
   [junit4]   2> ... 37 more
   [junit4]   2>
   [junit4]   2> 1157742 INFO  (qtp292962750-14213) [n:127.0.0.1:52862_solr c:test_col s:shard1  x:test_col_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=400 QTime=1739
   [junit4]   2> 1157742 ERROR (OverseerThreadFactory-5962-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:52862/solr
   [junit4]   2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:52862/solr: Error CREATEing SolrCore 'test_col_shard1_replica_n1': Unable to create core [test_col_shard1_replica_n1] Caused by: invalid API spec: apispec/emptySpec.json
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1157846 INFO  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1157846 INFO  (zkCallback-2751-thread-1-processing-n:127.0.0.1:52860_solr) [n:127.0.0.1:52860_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1157847 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1157871 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1157875 ERROR (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.c.CorePropertiesLocator Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties: Too many open files
   [junit4]   2> 1157879 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr c:test_col s:shard2 r:core_node1 x:test_col_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1157880 ERROR (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'test_col_shard2_replica_n1': Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:948)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:747)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.core.CorePropertiesLocator.writePropertiesFile(CorePropertiesLocator.java:97)
   [junit4]   2> at org.apache.solr.core.CorePropertiesLocator.create(CorePropertiesLocator.java:70)
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]   2> ... 34 more
   [junit4]   2>
   [junit4]   2> 1157881 INFO  (qtp76551027-14198) [n:127.0.0.1:52860_solr c:test_col s:shard2  x:test_col_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard2_replica_n1&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=400 QTime=1876
   [junit4]   2> 1157882 ERROR (OverseerThreadFactory-5962-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:52860/solr
   [junit4]   2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:52860/solr: Error CREATEing SolrCore 'test_col_shard2_replica_n1': Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node3/./test_col_shard2_replica_n1/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1157989 INFO  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1157989 INFO  (zkCallback-2753-thread-2-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1158185 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1158185 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1158185 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52858/solr/test_col_shard1_replica_n2/
   [junit4]   2> 1158185 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1158185 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:52858/solr/test_col_shard1_replica_n2/ has no replicas
   [junit4]   2> 1158185 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1158209 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52858/solr/test_col_shard1_replica_n2/ shard1
   [junit4]   2> 1158316 INFO  (zkCallback-2753-thread-2-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1158316 INFO  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1158319 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1158321 ERROR (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.c.CorePropertiesLocator Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties: Too many open files
   [junit4]   2> 1158325 ERROR (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'test_col_shard1_replica_n2': Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:948)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:747)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.core.CorePropertiesLocator.writePropertiesFile(CorePropertiesLocator.java:97)
   [junit4]   2> at org.apache.solr.core.CorePropertiesLocator.create(CorePropertiesLocator.java:70)
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]   2> ... 34 more
   [junit4]   2>
   [junit4]   2> 1158325 INFO  (qtp888409251-14187) [n:127.0.0.1:52858_solr c:test_col s:shard1  x:test_col_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-tlog.xml&property.schema=schema15.xml&name=test_col_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=400 QTime=2321
   [junit4]   2> 1158326 ERROR (OverseerThreadFactory-5962-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:52858/solr
   [junit4]   2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:52858/solr: Error CREATEing SolrCore 'test_col_shard1_replica_n2': Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node5/./test_col_shard1_replica_n2/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1158439 INFO  (zkCallback-2753-thread-2-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1160383 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr c:test_col s:shard2 r:core_node1 x:test_col_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52857/solr/test_col_shard2_replica_n2/
   [junit4]   2> 1160383 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr c:test_col s:shard2 r:core_node1 x:test_col_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1160383 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr c:test_col s:shard2 r:core_node1 x:test_col_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:52857/solr/test_col_shard2_replica_n2/ has no replicas
   [junit4]   2> 1160386 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr c:test_col s:shard2 r:core_node1 x:test_col_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52857/solr/test_col_shard2_replica_n2/ shard2
   [junit4]   2> 1160493 INFO  (zkCallback-2753-thread-1-processing-n:127.0.0.1:52857_solr) [n:127.0.0.1:52857_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1160720 INFO  (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1160723 ERROR (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.c.CorePropertiesLocator Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/./test_col_shard2_replica_n2/core.properties: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/./test_col_shard2_replica_n2/core.properties: Too many open files
   [junit4]   2> 1160727 ERROR (qtp1993358908-14192) [n:127.0.0.1:52857_solr c:test_col s:shard2  x:test_col_shard2_replica_n2] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'test_col_shard2_replica_n2': Couldn't persist core properties to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/./test_col_shard2_replica_n2/core.properties : /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudDeleteByQuery_4370360EC14CB8B1-001/tempDir-001/node1/./test_col_shard2_replica_n2/core.properties: Too many open files
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:948)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:747)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.serv

[...truncated too long message...]

   [junit4]   2> 1191898 INFO  (jetty-closer-2732-thread-5) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1191900 INFO  (zkCallback-2752-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1191900 INFO  (jetty-closer-2732-thread-5) [    ] o.a.s.c.Overseer Overseer (id=98152741432262667-127.0.0.1:52859_solr-n_0000000000) closing
   [junit4]   2> 1191901 INFO  (zkCallback-2751-thread-2-processing-n:127.0.0.1:52860_solr) [n:127.0.0.1:52860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1191901 INFO  (OverseerStateUpdate-98152741432262667-127.0.0.1:52859_solr-n_0000000000) [n:127.0.0.1:52859_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:52859_solr
   [junit4]   2> 1191903 INFO  (zkCallback-2752-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1193401 WARN  (zkCallback-2758-thread-2-processing-n:127.0.0.1:52862_solr) [n:127.0.0.1:52862_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1193401 WARN  (zkCallback-2757-thread-1-processing-n:127.0.0.1:52858_solr) [n:127.0.0.1:52858_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1193401 INFO  (jetty-closer-2732-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@68ad6c58{/solr,null,UNAVAILABLE}
   [junit4]   2> 1193401 INFO  (jetty-closer-2732-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@31447002{/solr,null,UNAVAILABLE}
   [junit4]   2> 1193404 WARN  (zkCallback-2751-thread-2-processing-n:127.0.0.1:52860_solr) [n:127.0.0.1:52860_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1193405 INFO  (jetty-closer-2732-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@52734ed9{/solr,null,UNAVAILABLE}
   [junit4]   2> 1193407 WARN  (zkCallback-2752-thread-1-processing-n:127.0.0.1:52859_solr) [n:127.0.0.1:52859_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1193407 INFO  (jetty-closer-2732-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7c9030b0{/solr,null,UNAVAILABLE}
   [junit4]   2> 1193409 ERROR (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1193410 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52855 52855
   [junit4]   2> 1193430 INFO  (Thread-3065) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52855 52855
   [junit4]   2> 1193667 WARN  (Thread-3065) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 6 /solr/aliases.json
   [junit4]   2> 6 /solr/clusterprops.json
   [junit4]   2> 5 /solr/security.json
   [junit4]   2> 3 /solr/configs/solrCloudCollectionConfig
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 6 /solr/clusterstate.json
   [junit4]   2> 4 /solr/overseer_elect/election/98152741432262667-127.0.0.1:52859_solr-n_0000000000
   [junit4]   2> 4 /solr/collections/test_col/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 6 /solr/live_nodes
   [junit4]   2> 6 /solr/collections
   [junit4]   2>
   [junit4]   2> 1193668 INFO  (SUITE-TestCloudDeleteByQuery-seed#[4370360EC14CB8B1]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Jun 17, 2017 9:33:36 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1627, maxMBSortInHeap=6.833646460789209, sim=RandomSimilarity(queryNorm=true): {}, locale=vi-VN, timezone=Etc/GMT+12
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=161997840,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [CopyFieldTest, SolrPluginUtilsTest, TestTrackingShardHandlerFactory, RAMDirectoryFactoryTest, StressHdfsTest, LeaderInitiatedRecoveryOnShardRestartTest, TestRequestForwarding, ClassificationUpdateProcessorTest, SolrXmlInZkTest, DocExpirationUpdateProcessorFactoryTest, DeleteShardTest, TestLegacyFieldCache, TestUpdate, TestWordDelimiterFilterFactory, TestDistributedStatsComponentCardinality, TestGraphTermsQParserPlugin, TestPerFieldSimilarityWithDefaultOverride, DistributedSuggestComponentTest, SimpleFacetsTest, TestSolrCloudWithDelegationTokens, MultiThreadedOCPTest, TestSolrJ, TestReloadDeadlock, OutOfBoxZkACLAndCredentialsProvidersTest, TestRuleBasedAuthorizationPlugin, AutoScalingHandlerTest, TestMinMaxOnMultiValuedField, AnalysisAfterCoreReloadTest, AnalyticsMergeStrategyTest, StandardRequestHandlerTest, HdfsCollectionsAPIDistributedZkTest, DateRangeFieldTest, TestStressLucene, TestSystemCollAutoCreate, SearchHandlerTest, BasicAuthStandaloneTest, DistribJoinFromCollectionTest, TestSystemIdResolver, SharedFSAutoReplicaFailoverUtilsTest, MultiTermTest, BooleanFieldTest, TestRestManager, ResponseHeaderTest, OpenExchangeRatesOrgProviderTest, TestDynamicFieldCollectionResource, TestSurroundQueryParser, TestStressVersions, VersionInfoTest, TestMacroExpander, TestSerializedLuceneMatchVersion, InfixSuggestersTest, CdcrReplicationHandlerTest, SolrMetricManagerTest, TestRandomCollapseQParserPlugin, IndexBasedSpellCheckerTest, TestNumericTerms64, WrapperMergePolicyFactoryTest, DirectoryFactoryTest, TestDocBasedVersionConstraints, TestSlowCompositeReaderWrapper, TestMiniSolrCloudCluster, TestRandomDVFaceting, CreateCollectionCleanupTest, SuggestComponentContextFilterQueryTest, TestCharFilters, PathHierarchyTokenizerFactoryTest, HdfsLockFactoryTest, TestLeaderElectionZkExpiry, HardAutoCommitTest, TestTolerantUpdateProcessorRandomCloud, TestFunctionQuery, TestExactSharedStatsCache, HighlighterMaxOffsetTest, DistributedMLTComponentTest, TestCollationFieldDocValues, HLLSerializationTest, TestManagedResource, TestMiniSolrCloudClusterSSL, TestRTimerTree, RecoveryZkTest, HdfsBasicDistributedZk2Test, DocValuesMultiTest, UUIDUpdateProcessorFallbackTest, DistributedVersionInfoTest, TestComplexPhraseQParserPlugin, TestExportWriter, TestLegacyNumericUtils, DocValuesTest, TestHighlightDedupGrouping, V2StandaloneTest, TestCollationField, SegmentsInfoRequestHandlerTest, TestSortByMinMaxFunction, BlockJoinFacetSimpleTest, IndexSchemaRuntimeFieldTest, TestFieldCache, DirectUpdateHandlerTest, TestFreeTextSuggestions, JSONWriterTest, TestBackupRepositoryFactory, PeerSyncTest, TestRawTransformer, TestBlendedInfixSuggestions, TestInitParams, TestFieldCacheSort, TestReplicationHandler, TestNamedUpdateProcessors, TestSolrDeletionPolicy1, PrimitiveFieldTypeTest, SpellCheckCollatorWithCollapseTest, TestSolrCoreSnapshots, SimplePostToolTest, TestSort, TestCollapseQParserPlugin, ShardSplitTest, SolrTestCaseJ4Test, TestSolrConfigHandlerCloud, SimpleMLTQParserTest, TestChildDocTransformer, ZkNodePropsTest, TestSolrCloudWithSecureImpersonation, TestBinaryResponseWriter, TestXIncludeConfig, SampleTest, TestRTGBase, TestConfigSetsAPI, MetricsConfigTest, SolrShardReporterTest, CurrencyFieldXmlFileTest, TestBinaryField, ClassificationUpdateProcessorFactoryTest, QueryElevationComponentTest, OverseerTest, TestSuggestSpellingConverter, DistributedQueryElevationComponentTest, TestBulkSchemaConcurrent, TestJmxIntegration, ConnectionReuseTest, ConfigureRecoveryStrategyTest, AliasIntegrationTest, DistributedFacetPivotSmallTest, MBeansHandlerTest, CleanupOldIndexTest, PingRequestHandlerTest, TestConfigSetsAPIZkFailure, DeleteNodeTest, AtomicUpdateProcessorFactoryTest, TestElisionMultitermQuery, TestOverriddenPrefixQueryForCustomFieldType, TestRandomFlRTGCloud, TestReRankQParserPlugin, TestCSVLoader, TestCloudDeleteByQuery]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCloudDeleteByQuery -Dtests.seed=4370360EC14CB8B1 -Dtests.slow=true -Dtests.locale=vi-VN -Dtests.timezone=Etc/GMT+12 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestCloudDeleteByQuery (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Could not find collection:test_col
   [junit4]    > at __randomizedtesting.SeedInfo.seed([4370360EC14CB8B1]:0)
   [junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
   [junit4]    > at org.apache.solr.cloud.TestCloudDeleteByQuery.createMiniSolrCloudCluster(TestCloudDeleteByQuery.java:117)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)Throwable #2: java.lang.NullPointerException
   [junit4]    > at __randomizedtesting.SeedInfo.seed([4370360EC14CB8B1]:0)
   [junit4]    > at org.apache.solr.cloud.TestCloudDeleteByQuery.afterClass(TestCloudDeleteByQuery.java:88)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [331/725 (1!)] on J0 in 39.73s, 0 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 44714 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4082 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4082/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([4D66C56555E5A3C6:18362DF7F91C6C36]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1153)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:836)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12518 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/init-core-data-001
   [junit4]   2> 2258873 WARN  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2258874 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 2258877 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2258877 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001
   [junit4]   2> 2258878 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2258878 INFO  (Thread-2768) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2258878 INFO  (Thread-2768) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2258882 ERROR (Thread-2768) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2258988 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:53491
   [junit4]   2> 2259014 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0000, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259015 INFO  (jetty-launcher-2403-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2259015 INFO  (jetty-launcher-2403-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2259018 INFO  (jetty-launcher-2403-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2259018 INFO  (jetty-launcher-2403-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2259059 INFO  (jetty-launcher-2403-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@298f3f15{/solr,null,AVAILABLE}
   [junit4]   2> 2259060 INFO  (jetty-launcher-2403-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5636aff3{HTTP/1.1,[http/1.1]}{127.0.0.1:53493}
   [junit4]   2> 2259060 INFO  (jetty-launcher-2403-thread-2) [    ] o.e.j.s.Server Started @2262354ms
   [junit4]   2> 2259061 INFO  (jetty-launcher-2403-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f240f78{/solr,null,AVAILABLE}
   [junit4]   2> 2259061 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53493}
   [junit4]   2> 2259061 ERROR (jetty-launcher-2403-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2259061 INFO  (jetty-launcher-2403-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@8b8d8f1{HTTP/1.1,[http/1.1]}{127.0.0.1:53494}
   [junit4]   2> 2259062 INFO  (jetty-launcher-2403-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@486f043a{/solr,null,AVAILABLE}
   [junit4]   2> 2259061 INFO  (jetty-launcher-2403-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6622a798{/solr,null,AVAILABLE}
   [junit4]   2> 2259062 INFO  (jetty-launcher-2403-thread-1) [    ] o.e.j.s.Server Started @2262356ms
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53494}
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2b6a4b47{HTTP/1.1,[http/1.1]}{127.0.0.1:53496}
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@170007aa{HTTP/1.1,[http/1.1]}{127.0.0.1:53495}
   [junit4]   2> 2259062 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-3) [    ] o.e.j.s.Server Started @2262357ms
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53495}
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T18:56:04.531Z
   [junit4]   2> 2259063 ERROR (jetty-launcher-2403-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2259063 INFO  (jetty-launcher-2403-thread-4) [    ] o.e.j.s.Server Started @2262357ms
   [junit4]   2> 2259064 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53496}
   [junit4]   2> 2259064 ERROR (jetty-launcher-2403-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2259063 ERROR (jetty-launcher-2403-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2259064 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2259065 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2259065 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2259065 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2259065 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2259064 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2259066 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2259066 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2259066 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T18:56:04.534Z
   [junit4]   2> 2259065 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T18:56:04.533Z
   [junit4]   2> 2259065 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2259066 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-17T18:56:04.534Z
   [junit4]   2> 2259069 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2259070 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2259070 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2259070 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2259083 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259085 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0004, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259087 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259088 INFO  (jetty-launcher-2403-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53491/solr
   [junit4]   2> 2259088 INFO  (jetty-launcher-2403-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53491/solr
   [junit4]   2> 2259090 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0003, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259094 INFO  (jetty-launcher-2403-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53491/solr
   [junit4]   2> 2259098 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259099 INFO  (jetty-launcher-2403-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53491/solr
   [junit4]   2> 2259101 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd0007, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259110 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cb76982fd000b, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2259166 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2259166 INFO  (jetty-launcher-2403-thread-2) [n:127.0.0.1:53493_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2259166 INFO  (jetty-launcher-2403-thread-3) [n:127.0.0.1:53495_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2259166 INFO  (jetty-launcher-2403-thread-1) [n:127.0.0.1:53494_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2259167 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53496_solr
   [junit4]   2> 2259168 INFO  (jetty-launcher-2403-thread-1) [n:127.0.0.1:53494_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53494_solr
   [junit4]   2> 2259169 INFO  (jetty-launcher-2403-thread-2) [n:127.0.0.1:53493_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53493_solr
   [junit4]   2> 2259169 INFO  (jetty-launcher-2403-thread-3) [n:127.0.0.1:53495_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53495_solr
   [junit4]   2> 2259170 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.c.Overseer Overseer (id=98154955692376074-127.0.0.1:53496_solr-n_0000000000) starting
   [junit4]   2> 2259172 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2259173 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2259174 INFO  (zkCallback-2420-thread-1-processing-n:127.0.0.1:53494_solr) [n:127.0.0.1:53494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2259174 INFO  (zkCallback-2424-thread-1-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2259175 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2259176 INFO  (zkCallback-2420-thread-1-processing-n:127.0.0.1:53494_solr) [n:127.0.0.1:53494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2259176 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2259177 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2259177 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2259209 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53496_solr
   [junit4]   2> 2259212 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2259212 INFO  (zkCallback-2420-thread-1-processing-n:127.0.0.1:53494_solr) [n:127.0.0.1:53494_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2259212 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2259212 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2259346 INFO  (jetty-launcher-2403-thread-3) [n:127.0.0.1:53495_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53495.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259353 INFO  (jetty-launcher-2403-thread-2) [n:127.0.0.1:53493_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53493.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259358 INFO  (jetty-launcher-2403-thread-2) [n:127.0.0.1:53493_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53493.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259358 INFO  (jetty-launcher-2403-thread-2) [n:127.0.0.1:53493_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53493.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259359 INFO  (jetty-launcher-2403-thread-2) [n:127.0.0.1:53493_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/.
   [junit4]   2> 2259374 INFO  (jetty-launcher-2403-thread-1) [n:127.0.0.1:53494_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53494.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259379 INFO  (jetty-launcher-2403-thread-1) [n:127.0.0.1:53494_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53494.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259379 INFO  (jetty-launcher-2403-thread-1) [n:127.0.0.1:53494_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53494.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259381 INFO  (jetty-launcher-2403-thread-1) [n:127.0.0.1:53494_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node1/.
   [junit4]   2> 2259382 INFO  (jetty-launcher-2403-thread-3) [n:127.0.0.1:53495_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53495.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259382 INFO  (jetty-launcher-2403-thread-3) [n:127.0.0.1:53495_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53495.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259383 INFO  (jetty-launcher-2403-thread-3) [n:127.0.0.1:53495_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/.
   [junit4]   2> 2259427 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53496.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259432 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53496.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259432 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53496.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2259433 INFO  (jetty-launcher-2403-thread-4) [n:127.0.0.1:53496_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/.
   [junit4]   2> 2259508 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2259510 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53491/solr ready
   [junit4]   2> 2259564 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[4D66C56555E5A3C6]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateShardRepFactor
   [junit4]   2> 2259566 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[4D66C56555E5A3C6]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2259574 INFO  (qtp289796548-15356) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2259577 INFO  (OverseerThreadFactory-6227-thread-1-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCreateShardRepFactor
   [junit4]   2> 2259689 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2259691 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2259691 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2259692 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2259802 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2259802 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2260708 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2260708 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2260727 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.IndexSchema [testCreateShardRepFactor_b_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2260727 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.IndexSchema [testCreateShardRepFactor_a_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2260752 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2260752 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_a_replica_n1' using configuration from collection testCreateShardRepFactor, trusted=true
   [junit4]   2> 2260752 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2260752 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_b_replica_n1' using configuration from collection testCreateShardRepFactor, trusted=true
   [junit4]   2> 2260752 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53496.solr.core.testCreateShardRepFactor.a.replica_n1' (registry 'solr.core.testCreateShardRepFactor.a.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2260752 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2260752 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrCore [[testCreateShardRepFactor_a_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/testCreateShardRepFactor_a_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/./testCreateShardRepFactor_a_replica_n1/data/]
   [junit4]   2> 2260752 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53493.solr.core.testCreateShardRepFactor.b.replica_n1' (registry 'solr.core.testCreateShardRepFactor.b.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2260753 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2260753 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrCore [[testCreateShardRepFactor_b_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/testCreateShardRepFactor_b_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/./testCreateShardRepFactor_b_replica_n1/data/]
   [junit4]   2> 2260806 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2260806 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2260806 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2260806 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2260807 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2260807 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2260807 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2260808 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2260809 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a29d441[testCreateShardRepFactor_b_replica_n1] main]
   [junit4]   2> 2260809 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5c602d57[testCreateShardRepFactor_a_replica_n1] main]
   [junit4]   2> 2260811 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2260811 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2260811 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2260811 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2260811 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2260812 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2260812 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570479293102817280
   [junit4]   2> 2260812 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570479293102817280
   [junit4]   2> 2260812 INFO  (searcherExecutor-6236-thread-1-processing-n:127.0.0.1:53496_solr x:testCreateShardRepFactor_a_replica_n1 s:a c:testCreateShardRepFactor) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica_n1] Registered new searcher Searcher@5c602d57[testCreateShardRepFactor_a_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2260813 INFO  (searcherExecutor-6237-thread-1-processing-n:127.0.0.1:53493_solr x:testCreateShardRepFactor_b_replica_n1 s:b c:testCreateShardRepFactor) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica_n1] Registered new searcher Searcher@3a29d441[testCreateShardRepFactor_b_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2260819 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2260819 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2260819 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2260819 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2260819 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53493/solr/testCreateShardRepFactor_b_replica_n1/
   [junit4]   2> 2260819 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53496/solr/testCreateShardRepFactor_a_replica_n1/
   [junit4]   2> 2260819 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2260819 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2260819 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:53493/solr/testCreateShardRepFactor_b_replica_n1/ has no replicas
   [junit4]   2> 2260819 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:53496/solr/testCreateShardRepFactor_a_replica_n1/ has no replicas
   [junit4]   2> 2260819 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2260819 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2260824 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53493/solr/testCreateShardRepFactor_b_replica_n1/ b
   [junit4]   2> 2260824 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53496/solr/testCreateShardRepFactor_a_replica_n1/ a
   [junit4]   2> 2260930 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2260930 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2260981 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2260981 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2260984 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1295
   [junit4]   2> 2260984 INFO  (qtp1848292336-15371) [n:127.0.0.1:53496_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1293
   [junit4]   2> 2260987 INFO  (qtp289796548-15356) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2261098 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2261098 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2261589 INFO  (OverseerCollectionConfigSetProcessor-98154955692376074-127.0.0.1:53496_solr-n_0000000000) [n:127.0.0.1:53496_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2261998 INFO  (qtp289796548-15356) [n:127.0.0.1:53493_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2} status=0 QTime=2424
   [junit4]   2> 2261999 INFO  (qtp289796548-15358) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createshard with params action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2262007 INFO  (OverseerThreadFactory-6227-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.CreateShardCmd Create shard invoked: {
   [junit4]   2>   "collection":"testCreateShardRepFactor",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "operation":"createshard"}
   [junit4]   2> 2262138 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2262138 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2262234 INFO  (OverseerThreadFactory-6227-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.CreateShardCmd Creating replica testCreateShardRepFactor_x_replica_n1 as part of slice x of collection testCreateShardRepFactor on 127.0.0.1:53495_solr
   [junit4]   2> 2262236 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&name=testCreateShardRepFactor_x_replica_n1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2262237 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2262347 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2262347 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2262347 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2263257 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2263263 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.IndexSchema [testCreateShardRepFactor_x_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2263294 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2263294 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_x_replica_n1' using configuration from collection testCreateShardRepFactor, trusted=true
   [junit4]   2> 2263295 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53495.solr.core.testCreateShardRepFactor.x.replica_n1' (registry 'solr.core.testCreateShardRepFactor.x.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2263295 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2263295 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrCore [[testCreateShardRepFactor_x_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/testCreateShardRepFactor_x_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/./testCreateShardRepFactor_x_replica_n1/data/]
   [junit4]   2> 2263370 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2263370 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2263373 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2263373 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2263374 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@60ccfd0a[testCreateShardRepFactor_x_replica_n1] main]
   [junit4]   2> 2263376 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2263377 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2263378 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2263378 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570479295793463296
   [junit4]   2> 2263379 INFO  (searcherExecutor-6246-thread-1-processing-n:127.0.0.1:53495_solr x:testCreateShardRepFactor_x_replica_n1 s:x c:testCreateShardRepFactor) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica_n1] Registered new searcher Searcher@60ccfd0a[testCreateShardRepFactor_x_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2263386 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2263386 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2263386 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53495/solr/testCreateShardRepFactor_x_replica_n1/
   [junit4]   2> 2263386 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2263386 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:53495/solr/testCreateShardRepFactor_x_replica_n1/ has no replicas
   [junit4]   2> 2263386 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2263391 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53495/solr/testCreateShardRepFactor_x_replica_n1/ x
   [junit4]   2> 2263501 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2263501 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2263501 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2263551 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2263553 INFO  (qtp897135131-15366) [n:127.0.0.1:53495_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&name=testCreateShardRepFactor_x_replica_n1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1317
   [junit4]   2> 2263554 INFO  (OverseerThreadFactory-6227-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.CreateShardCmd Finished create command on all shards for collection: testCreateShardRepFactor
   [junit4]   2> 2263556 INFO  (qtp289796548-15358) [n:127.0.0.1:53493_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2} status=0 QTime=1556
   [junit4]   2> 2263557 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[4D66C56555E5A3C6]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateShardRepFactor
   [junit4]   2> 2263590 INFO  (TEST-CustomCollectionTest.testRouteFieldForHashRouter-seed#[4D66C56555E5A3C6]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRouteFieldForHashRouter
   [junit4]   2> 2263592 INFO  (TEST-CustomCollectionTest.testRouteFieldForHashRouter-seed#[4D66C56555E5A3C6]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2263595 INFO  (qtp289796548-15359) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2263599 INFO  (OverseerThreadFactory-6227-thread-3-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 2263600 INFO  (qtp289796548-15361) [n:127.0.0.1:53493_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.b.replica_n1, tag=null
   [junit4]   2> 2263600 INFO  (qtp1848292336-15372) [n:127.0.0.1:53496_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.a.replica_n1, tag=null
   [junit4]   2> 2263603 INFO  (qtp897135131-15367) [n:127.0.0.1:53495_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.x.replica_n1, tag=null
   [junit4]   2> 2263604 INFO  (OverseerCollectionConfigSetProcessor-98154955692376074-127.0.0.1:53496_solr-n_0000000000) [n:127.0.0.1:53496_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2263642 INFO  (qtp289796548-15361) [n:127.0.0.1:53493_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4ad7d0ca
   [junit4]   2> 2263645 INFO  (qtp1848292336-15372) [n:127.0.0.1:53496_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@64fdd572
   [junit4]   2> 2263645 INFO  (qtp289796548-15361) [n:127.0.0.1:53493_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.b.replica_n1, tag=1255657674
   [junit4]   2> 2263645 INFO  (qtp289796548-15361) [n:127.0.0.1:53493_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCreateShardRepFactor.b.leader, tag=1255657674
   [junit4]   2> 2263647 INFO  (qtp897135131-15367) [n:127.0.0.1:53495_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6f4489c7
   [junit4]   2> 2263649 INFO  (qtp1848292336-15372) [n:127.0.0.1:53496_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.a.replica_n1, tag=1694356850
   [junit4]   2> 2263650 INFO  (qtp1848292336-15372) [n:127.0.0.1:53496_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCreateShardRepFactor.a.leader, tag=1694356850
   [junit4]   2> 2263650 INFO  (qtp897135131-15367) [n:127.0.0.1:53495_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.x.replica_n1, tag=1866762695
   [junit4]   2> 2263650 INFO  (qtp897135131-15367) [n:127.0.0.1:53495_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCreateShardRepFactor.x.leader, tag=1866762695
   [junit4]   2> 2263652 INFO  (qtp289796548-15361) [n:127.0.0.1:53493_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_b_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=51
   [junit4]   2> 2263657 INFO  (qtp897135131-15367) [n:127.0.0.1:53495_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_x_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=55
   [junit4]   2> 2263658 INFO  (qtp1848292336-15372) [n:127.0.0.1:53496_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_a_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=57
   [junit4]   2> 2264386 INFO  (qtp289796548-15359) [n:127.0.0.1:53493_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2} status=0 QTime=791
   [junit4]   2> 2264387 INFO  (qtp289796548-15363) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=3&router.field=shard_s&name=routeFieldColl&nrtReplicas=2&action=CREATE&numShards=4&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2264398 INFO  (OverseerThreadFactory-6227-thread-4-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.CreateCollectionCmd Create collection routeFieldColl
   [junit4]   2> 2264403 INFO  (OverseerCollectionConfigSetProcessor-98154955692376074-127.0.0.1:53496_solr-n_0000000000) [n:127.0.0.1:53496_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2264619 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard4_replica_n1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard4&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264620 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard2_replica_n1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264622 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard3_replica_n2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264623 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard1_replica_n1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264623 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard1_replica_n2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264624 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard3_replica_n1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264625 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard2_replica_n2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264625 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2264627 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard4_replica_n2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard4&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2264747 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2264748 INFO  (zkCallback-2422-thread-1-processing-n:127.0.0.1:53495_solr) [n:127.0.0.1:53495_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2264748 INFO  (zkCallback-2420-thread-1-processing-n:127.0.0.1:53494_solr) [n:127.0.0.1:53494_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2264748 INFO  (zkCallback-2424-thread-2-processing-n:127.0.0.1:53496_solr) [n:127.0.0.1:53496_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2265664 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265664 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265720 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265720 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265725 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265734 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265737 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 2265740 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2265742 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 2265762 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265762 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2265768 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 2265799 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2265800 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 2265810 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2265811 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2265850 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265850 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard4_replica_n2' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265851 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53494.solr.core.routeFieldColl.shard4.replica_n2' (registry 'solr.core.routeFieldColl.shard4.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265851 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265851 INFO  (qtp1567084168-15380) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n2] o.a.s.c.SolrCore [[routeFieldColl_shard4_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node1/routeFieldColl_shard4_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node1/./routeFieldColl_shard4_replica_n2/data/]
   [junit4]   2> 2265857 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265857 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard1_replica_n2' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265891 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265891 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard1_replica_n1' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265892 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53496.solr.core.routeFieldColl.shard1.replica_n1' (registry 'solr.core.routeFieldColl.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265892 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265892 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.c.SolrCore [[routeFieldColl_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/routeFieldColl_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/./routeFieldColl_shard1_replica_n1/data/]
   [junit4]   2> 2265894 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265894 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard2_replica_n2' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265894 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53494.solr.core.routeFieldColl.shard2.replica_n2' (registry 'solr.core.routeFieldColl.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265894 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265894 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.c.SolrCore [[routeFieldColl_shard2_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node1/routeFieldColl_shard2_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node1/./routeFieldColl_shard2_replica_n2/data/]
   [junit4]   2> 2265896 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265896 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard3_replica_n1' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265896 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53496.solr.core.routeFieldColl.shard3.replica_n1' (registry 'solr.core.routeFieldColl.shard3.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265897 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265897 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.c.SolrCore [[routeFieldColl_shard3_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/routeFieldColl_shard3_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node4/./routeFieldColl_shard3_replica_n1/data/]
   [junit4]   2> 2265898 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265898 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard2_replica_n1' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265859 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53495.solr.core.routeFieldColl.shard1.replica_n2' (registry 'solr.core.routeFieldColl.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265898 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53493.solr.core.routeFieldColl.shard2.replica_n1' (registry 'solr.core.routeFieldColl.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265898 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265898 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.c.SolrCore [[routeFieldColl_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/routeFieldColl_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/./routeFieldColl_shard1_replica_n2/data/]
   [junit4]   2> 2265900 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265902 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard4_replica_n1' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265902 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2265903 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard3_replica_n2' using configuration from collection routeFieldColl, trusted=true
   [junit4]   2> 2265903 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53493.solr.core.routeFieldColl.shard4.replica_n1' (registry 'solr.core.routeFieldColl.shard4.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265903 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53495.solr.core.routeFieldColl.shard3.replica_n2' (registry 'solr.core.routeFieldColl.shard3.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bbd52c7
   [junit4]   2> 2265903 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265903 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265903 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.c.SolrCore [[routeFieldColl_shard4_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/routeFieldColl_shard4_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/./routeFieldColl_shard4_replica_n1/data/]
   [junit4]   2> 2265898 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2265903 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.c.SolrCore [[routeFieldColl_shard2_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/routeFieldColl_shard2_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node2/./routeFieldColl_shard2_replica_n1/data/]
   [junit4]   2> 2265903 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.c.SolrCore [[routeFieldColl_shard3_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/routeFieldColl_shard3_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001/tempDir-001/node3/./routeFieldColl_shard3_replica_n2/data/]
   [junit4]   2> 2266068 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266068 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266068 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266070 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2266070 INFO  (qtp1848292336-15373) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2266068 INFO  (qtp897135131-15368) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266080 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266080 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266082 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2266082 INFO  (qtp289796548-15354) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2266086 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266086 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266086 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266086 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266087 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266087 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266088 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2266088 INFO  (qtp1848292336-15374) [n:127.0.0.1:53496_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2266089 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2266089 INFO  (qtp289796548-15357) [n:127.0.0.1:53493_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2266090 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2266090 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2266090 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2266090 INFO  (qtp1567084168-15360) [n:127.0.0.1:53494_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2266091 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2266092 INFO  (qtp897135131-15369) [n:127.0.0.1:53495_solr c:routeFieldColl s:shard1  x:routeFieldColl_sha

[...truncated too long message...]

eader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 2313184 INFO  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53493_solr
   [junit4]   2> 2313186 INFO  (jetty-closer-2404-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6622a798{/solr,null,UNAVAILABLE}
   [junit4]   2> 2313194 INFO  (coreCloseExecutor-6425-thread-4) [n:127.0.0.1:53493_solr c:implicitcoll s:c r:core_node15 x:implicitcoll_c_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.implicitcoll.c.leader, tag=1177393629
   [junit4]   2> 2313194 INFO  (coreCloseExecutor-6425-thread-2) [n:127.0.0.1:53493_solr c:implicitcoll s:b r:core_node6 x:implicitcoll_b_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.implicitcoll.b.leader, tag=1283046151
   [junit4]   2> 2313194 INFO  (coreCloseExecutor-6426-thread-5) [n:127.0.0.1:53494_solr c:implicitcoll s:x r:core_node16 x:implicitcoll_x_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.implicitcoll.x.leader, tag=1948239832
   [junit4]   2> 2313198 WARN  (zkCallback-2420-thread-1-processing-n:127.0.0.1:53494_solr) [n:127.0.0.1:53494_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2313198 INFO  (jetty-closer-2404-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1f240f78{/solr,null,UNAVAILABLE}
   [junit4]   2> 2314700 WARN  (zkCallback-2425-thread-1-processing-n:127.0.0.1:53493_solr) [n:127.0.0.1:53493_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2314701 INFO  (jetty-closer-2404-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@298f3f15{/solr,null,UNAVAILABLE}
   [junit4]   2> 2314713 ERROR (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2314714 INFO  (SUITE-CustomCollectionTest-seed#[4D66C56555E5A3C6]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53491 53491
   [junit4]   2> 2314817 INFO  (Thread-2768) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53491 53491
   [junit4]   2> 2314933 WARN  (Thread-2768) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 11 /solr/configs/conf
   [junit4]   2> 9 /solr/clusterprops.json
   [junit4]   2> 9 /solr/aliases.json
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 9 /solr/clusterstate.json
   [junit4]   2> 5 /solr/collections/implicitcoll/state.json
   [junit4]   2> 4 /solr/collections/testCreateShardRepFactor/state.json
   [junit4]   2> 4 /solr/collections/routeFieldColl/state.json
   [junit4]   2> 4 /solr/collections/withShardField/state.json
   [junit4]   2> 2 /solr/collections/implicitcoll/leader_elect/x/election/98154955692376072-core_node16-n_0000000000
   [junit4]   2> 2 /solr/collections/implicitcoll/leader_elect/c/election/98154955692376072-core_node13-n_0000000001
   [junit4]   2> 2 /solr/overseer_elect/election/98154955692376076-127.0.0.1:53493_solr-n_0000000001
   [junit4]   2> 2 /solr/collections/implicitcoll/leader_elect/x/election/98154955692376076-core_node19-n_0000000002
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 27 /solr/collections
   [junit4]   2> 9 /solr/live_nodes
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_4D66C56555E5A3C6-001
   [junit4]   2> Jun 17, 2017 6:57:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=el, timezone=Antarctica/McMurdo
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=127565736,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestDFISimilarityFactory, TestBinaryResponseWriter, HighlighterMaxOffsetTest, TestXmlQParser, CursorPagingTest, CollectionReloadTest, FastVectorHighlighterTest, BlockJoinFacetDistribTest, TestSchemaSimilarityResource, HighlighterTest, TestFieldCache, CdcrBootstrapTest, TestBinaryField, TestStressVersions, BasicAuthStandaloneTest, TestLRUStatsCache, SpatialFilterTest, PeerSyncTest, DocExpirationUpdateProcessorFactoryTest, HLLSerializationTest, TestHdfsCloudBackupRestore, TestManagedSynonymGraphFilterFactory, SOLR749Test, TestSolrXml, LeaderElectionContextKeyTest, TestMaxScoreQueryParser, DeleteShardTest, TestCSVResponseWriter, HdfsUnloadDistributedZkTest, BooleanFieldTest, TestStressUserVersions, SolrCLIZkUtilsTest, TestIBSimilarityFactory, TestRawTransformer, StandardRequestHandlerTest, ClassificationUpdateProcessorTest, TestSolrCloudWithHadoopAuthPlugin, TestFreeTextSuggestions, TestFieldTypeCollectionResource, TestExtendedDismaxParser, PrimitiveFieldTypeTest, TestConfigsApi, SolrRequestParserTest, MissingSegmentRecoveryTest, ResponseHeaderTest, TestApiFramework, EnumFieldTest, TestCloudDeleteByQuery, TestRestManager, UnloadDistributedZkTest, ForceLeaderTest, TestPerFieldSimilarityWithDefaultOverride, DistributedQueryComponentCustomSortTest, HdfsChaosMonkeyNothingIsSafeTest, TestLegacyNumericUtils, TestTrackingShardHandlerFactory, OverseerRolesTest, SimpleFacetsTest, TestSortByMinMaxFunction, DistributedFacetPivotSmallTest, TestReqParamsAPI, SolrShardReporterTest, ZkNodePropsTest, CreateCollectionCleanupTest, TestStressReorder, SharedFSAutoReplicaFailoverUtilsTest, RequiredFieldsTest, TestUpdate, DeleteInactiveReplicaTest, TestDelegationWithHadoopAuth, TestSolrDeletionPolicy1, DistributedDebugComponentTest, TestFieldCacheSort, PingRequestHandlerTest, TestTestInjection, TestRandomDVFaceting, TestManagedStopFilterFactory, BinaryUpdateRequestHandlerTest, TestPushWriter, ClusterStateUpdateTest, TestFaceting, TermsComponentTest, TestFieldCacheVsDocValues, TestMacroExpander, FileUtilsTest, TestStressLucene, NumericFieldsTest, DirectSolrConnectionTest, TriLevelCompositeIdRoutingTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestMaxTokenLenTokenizer, DocValuesTest, TestCustomStream, DateFieldTest, SolrCoreTest, TestClusterProperties, TestCloudNestedDocsSort, SyncSliceTest, XmlUpdateRequestHandlerTest, IndexSchemaRuntimeFieldTest, TestBlendedInfixSuggestions, BigEndianAscendingWordSerializerTest, SampleTest, TestEmbeddedSolrServerSchemaAPI, TestHttpShardHandlerFactory, RestartWhileUpdatingTest, TestDocBasedVersionConstraints, TestRTimerTree, TestMergePolicyConfig, TestSimpleTextCodec, AddBlockUpdateTest, TestDownShardTolerantSearch, TestRebalanceLeaders, RAMDirectoryFactoryTest, DistributedQueryElevationComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestMultiValuedNumericRangeQuery, TestCollationField, TestRuleBasedAuthorizationPlugin, LeaderElectionTest, ExplicitHLLTest, TestOverriddenPrefixQueryForCustomFieldType, TestSolrJ, TestRandomFlRTGCloud, TestDynamicFieldCollectionResource, TestSort, HardAutoCommitTest, SolrMetricManagerTest, TestLocalFSCloudBackupRestore, DateRangeFieldTest, SolrMetricsIntegrationTest, CdcrUpdateLogTest, ZkStateReaderTest, TestShortCircuitedRequests, TestAuthorizationFramework, TestSolr4Spatial2, TestDeleteCollectionOnDownNodes, PeerSyncWithIndexFingerprintCachingTest, TestLeaderElectionZkExpiry, TestComplexPhraseQParserPlugin, PropertiesRequestHandlerTest, TestSystemIdResolver, TestRequestForwarding, TestJmxIntegration, MergeStrategyTest, RecoveryZkTest, TestRandomCollapseQParserPlugin, CdcrVersionReplicationTest, TestLegacyField, BlockJoinFacetSimpleTest, TestFunctionQuery, OverseerCollectionConfigSetProcessorTest, HdfsCollectionsAPIDistributedZkTest, StressHdfsTest, AtomicUpdateProcessorFactoryTest, TestSimpleQParserPlugin, HdfsRecoverLeaseTest, TestUseDocValuesAsStored, CurrencyFieldXmlFileTest, TestLFUCache, UUIDUpdateProcessorFallbackTest, AlternateDirectoryTest, PathHierarchyTokenizerFactoryTest, XsltUpdateRequestHandlerTest, TestSerializedLuceneMatchVersion, DistributedFacetExistsSmallTest, TestBM25SimilarityFactory, TestCollationFieldDocValues, InfixSuggestersTest, AnalyticsMergeStrategyTest, TestPartialUpdateDeduplication, TestBulkSchemaConcurrent, MultiTermTest, HdfsLockFactoryTest, ZkControllerTest, DeleteNodeTest, QueryElevationComponentTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, TestZkChroot, ZkCLITest, TestDistributedSearch, TestHashPartitioner, TestJoin, DistributedTermsComponentTest, TestCoreContainer, StatsComponentTest, DirectUpdateHandlerTest, TestBadConfig, SoftAutoCommitTest, TestQueryUtils, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TestQueryTypes, DocumentBuilderTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, RegexBoostProcessorTest, MBeansHandlerTest, TestAnalyzedSuggestions, JsonLoaderTest, HighlighterConfigTest, UpdateParamsTest, TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, TestStressRecovery, TestTolerantSearch, ActionThrottleTest, AliasIntegrationTest, CleanupOldIndexTest, CollectionStateFormat2Test, CollectionTooManyReplicasTest, ConfigSetsAPITest, CustomCollectionTest]
   [junit4] Completed [503/725 (1!)] on J0 in 56.85s, 4 tests, 1 error <<< FAILURES!

[...truncated 44179 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4083 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4083/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [Overseer] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.Overseer.start(Overseer.java:527)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:774)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)  at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1777)  at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [Overseer]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.Overseer.start(Overseer.java:527)
        at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:774)
        at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
        at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1777)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([CF3593D6485D48D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:295)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ZkControllerTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:     1) Thread[id=16555, name=OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=16557, name=OverseerHdfsCoreFailoverThread-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:
   1) Thread[id=16555, name=OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer state updater.]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
        at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=16557, name=OverseerHdfsCoreFailoverThread-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([CF3593D6485D48D]:0)




Build Log:
[...truncated 12164 lines...]
   [junit4] Suite: org.apache.solr.cloud.ZkControllerTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ZkControllerTest_CF3593D6485D48D-001/init-core-data-001
   [junit4]   2> 1621784 WARN  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=23 numCloses=23
   [junit4]   2> 1621784 INFO  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1621788 INFO  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1621790 INFO  (TEST-ZkControllerTest.testNodeNameUrlConversion-seed#[CF3593D6485D48D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeNameUrlConversion
   [junit4]   2> 1621791 INFO  (TEST-ZkControllerTest.testNodeNameUrlConversion-seed#[CF3593D6485D48D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1621791 INFO  (Thread-2992) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1621791 INFO  (Thread-2992) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1621795 ERROR (Thread-2992) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1621893 INFO  (TEST-ZkControllerTest.testNodeNameUrlConversion-seed#[CF3593D6485D48D]) [    ] o.a.s.c.ZkTestServer start zk server on port:51255
   [junit4]   2> 1622022 ERROR (TEST-ZkControllerTest.testNodeNameUrlConversion-seed#[CF3593D6485D48D]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1622022 INFO  (TEST-ZkControllerTest.testNodeNameUrlConversion-seed#[CF3593D6485D48D]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51255 51255
   [junit4]   2> 1622043 INFO  (Thread-2992) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51255 51255
   [junit4]   2> 1622273 WARN  (Thread-2992) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/aliases.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/clusterstate.json
   [junit4]   2> 2 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/live_nodes
   [junit4]   2> 2 /solr/collections
   [junit4]   2>
   [junit4]   2> 1622274 INFO  (TEST-ZkControllerTest.testNodeNameUrlConversion-seed#[CF3593D6485D48D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeNameUrlConversion
   [junit4] IGNOR/A 0.00s J0 | ZkControllerTest.testPublishAndWaitForDownStates
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/SOLR-7736))
   [junit4]   2> 1622276 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testReadConfigName
   [junit4]   2> 1622276 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1622277 INFO  (Thread-2993) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1622277 INFO  (Thread-2993) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1622279 ERROR (Thread-2993) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1622380 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [    ] o.a.s.c.ZkTestServer start zk server on port:51259
   [junit4]   2> 1622420 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6a1d78b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1622461 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1622462 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:8983_solr
   [junit4]   2> 1622464 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=98157001021259779-127.0.0.1:8983_solr-n_0000000000) starting
   [junit4]   2> 1622481 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 1622483 INFO  (zkCallback-2675-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1622487 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=98157001021259779-127.0.0.1:8983_solr-n_0000000000) closing
   [junit4]   2> 1622496 INFO  (OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000000) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:8983_solr
   [junit4]   2> 1622497 WARN  (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer I'm exiting, but I'm still the leader
   [junit4]   2> 1622501 INFO  (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:8983_solr
   [junit4]   2> 1622502 INFO  (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=98157001021259779-127.0.0.1:8983_solr-n_0000000001) starting
   [junit4]   2> 1622517 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=477747794
   [junit4]   2> 1622517 ERROR (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1622518 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51259 51259
   [junit4]   2> 1622533 INFO  (Thread-2993) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51259 51259
   [junit4]   2> 1623033 WARN  (Thread-2993) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> 1623033 INFO  (TEST-ZkControllerTest.testReadConfigName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 ###Ending testReadConfigName
   [junit4]   2> 1623036 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 ###Starting testGetHostName
   [junit4]   2> 1623037 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1623037 INFO  (Thread-2994) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1623037 INFO  (Thread-2994) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1623038 ERROR (Thread-2994) [n:127.0.0.1:8983_solr    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1623147 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer start zk server on port:51264
   [junit4]   2> 1623169 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6a1d78b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1623203 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1623204 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:8983_solr
   [junit4]   2> 1623206 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=98157001071067138-127.0.0.1:8983_solr-n_0000000000) starting
   [junit4]   2> 1623220 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 1623223 INFO  (zkCallback-2681-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1623223 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer (id=98157001071067138-127.0.0.1:8983_solr-n_0000000000) closing
   [junit4]   2> 1623223 INFO  (OverseerStateUpdate-98157001071067138-127.0.0.1:8983_solr-n_0000000000) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:8983_solr
   [junit4]   2> 1623225 WARN  (zkCallback-2681-thread-1-processing-n:127.0.0.1:8983_solr) [n:127.0.0.1:8983_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1623225 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=207893525
   [junit4]   2> 1623225 ERROR (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1623226 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51264 51264
   [junit4]   2> 1623245 INFO  (Thread-2994) [n:127.0.0.1:8983_solr    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51264 51264
   [junit4]   2> 1623989 INFO  (TEST-ZkControllerTest.testGetHostName-seed#[CF3593D6485D48D]) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 ###Ending testGetHostName
   [junit4]   2> 1623991 INFO  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 Waiting for all tracked resources to be released
   [junit4]   2> 1624514 INFO  (OverseerCollectionConfigSetProcessor-98157001021259779-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=98157001021259779-127.0.0.1:8983_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 1624729 ERROR (OverseerExitThread) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:291)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1634050 INFO  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 Waiting for all tracked resources to be released
   [junit4]   2> 1644091 INFO  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 Waiting for all tracked resources to be released
   [junit4]   2> JUnit4-serializer-daemon:
   [junit4]   2>     java.lang.Thread.sleep(Native Method)
   [junit4]   2>     com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>
   [junit4]   2> Reference Handler:
   [junit4]   2>     java.lang.Object.wait(Native Method)
   [junit4]   2>     java.lang.Object.wait(Object.java:502)
   [junit4]   2>     java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   [junit4]   2>     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
   [junit4]   2>
   [junit4]   2> main:
   [junit4]   2>     java.lang.Object.wait(Native Method)
   [junit4]   2>     java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>     java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>     [...com.carrotsearch.randomizedtesting.*]
   [junit4]   2>     com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2>     com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2>     com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>
   [junit4]   2> SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker:
   [junit4]   2>     java.lang.Thread.dumpThreads(Native Method)
   [junit4]   2>     java.lang.Thread.getAllStackTraces(Thread.java:1610)
   [junit4]   2>     org.apache.solr.SolrTestCaseJ4.clearObjectTrackerAndCheckEmpty(SolrTestCaseJ4.java:350)
   [junit4]   2>     org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:294)
   [junit4]   2>     [...sun.*, java.lang.reflect.*, com.carrotsearch.randomizedtesting.*]
   [junit4]   2>     org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>     [...com.carrotsearch.randomizedtesting.*]
   [junit4]   2>     org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>     [...com.carrotsearch.randomizedtesting.*]
   [junit4]   2>     org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>     org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>     org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>     org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>     [...com.carrotsearch.randomizedtesting.*]
   [junit4]   2>     java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> TimeLimitedCollector timer thread:
   [junit4]   2>     java.lang.Thread.sleep(Native Method)
   [junit4]   2>     org.apache.lucene.search.TimeLimitingCollector$TimerThread.run(TimeLimitingCollector.java:261)
   [junit4]   2>
   [junit4]   2> Signal Dispatcher:
   [junit4]   2>
   [junit4]   2> OverseerHdfsCoreFailoverThread-98157001021259779-127.0.0.1:8983_solr-n_0000000001:
   [junit4]   2>     java.lang.Thread.sleep(Native Method)
   [junit4]   2>     org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]   2>     java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001:
   [junit4]   2>     [...sun.*]
   [junit4]   2>     java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>     org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
   [junit4]   2>     org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)
   [junit4]   2>     org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)
   [junit4]   2>     java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> SUITE-ZkControllerTest-seed#[CF3593D6485D48D]:
   [junit4]   2>     java.lang.Object.wait(Native Method)
   [junit4]   2>     java.lang.Thread.join(Thread.java:1260)
   [junit4]   2>     [...com.carrotsearch.randomizedtesting.*]
   [junit4]   2>
   [junit4]   2> Finalizer:
   [junit4]   2>     java.lang.Object.wait(Native Method)
   [junit4]   2>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   [junit4]   2>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   [junit4]   2>     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
   [junit4]   2>
   [junit4]   2> 1645094 INFO  (SUITE-ZkControllerTest-seed#[CF3593D6485D48D]-worker) [n:127.0.0.1:8983_solr    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Jun 18, 2017 3:36:36 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Jun 18, 2017 3:36:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:
   [junit4]   2>    1) Thread[id=16555, name=OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer state updater.]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
   [junit4]   2>         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)
   [junit4]   2>         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    2) Thread[id=16557, name=OverseerHdfsCoreFailoverThread-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Jun 18, 2017 3:36:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16555, name=OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer state updater.]
   [junit4]   2>    2) Thread[id=16557, name=OverseerHdfsCoreFailoverThread-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2> 1665114 INFO  (OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001) [n:127.0.0.1:8983_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:8983_solr
   [junit4]   2> Jun 18, 2017 3:36:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=hi, timezone=Asia/Dushanbe
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=120186536,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestLegacyFieldCache, TestCursorMarkWithoutUniqueKey, PathHierarchyTokenizerFactoryTest, DirectoryFactoryTest, CdcrUpdateLogTest, TestSlowCompositeReaderWrapper, TestSort, TestCloudJSONFacetJoinDomain, HLLSerializationTest, DateMathParserTest, HardAutoCommitTest, DocumentAnalysisRequestHandlerTest, TestHdfsCloudBackupRestore, TestCoreContainer, ClassificationUpdateProcessorTest, ImplicitSnitchTest, TestStressVersions, SolrShardReporterTest, TestJmxIntegration, TestHttpShardHandlerFactory, DistributedFacetPivotSmallTest, TestReqParamsAPI, AnalysisAfterCoreReloadTest, TestMaxScoreQueryParser, SuggesterFSTTest, TestBinaryField, TestCollationFieldDocValues, DistributedQueryElevationComponentTest, TestLeaderElectionZkExpiry, TestXmlQParser, TestFieldCollectionResource, AliasIntegrationTest, TestManagedSchemaThreadSafety, WrapperMergePolicyFactoryTest, TestConfigsApi, BigEndianAscendingWordSerializerTest, OutOfBoxZkACLAndCredentialsProvidersTest, SpellCheckCollatorWithCollapseTest, HdfsChaosMonkeyNothingIsSafeTest, SolrCLIZkUtilsTest, CdcrReplicationHandlerTest, DistributedDebugComponentTest, TestDocBasedVersionConstraints, TestLRUStatsCache, DistributedFacetExistsSmallTest, ParsingFieldUpdateProcessorsTest, RestartWhileUpdatingTest, BlockJoinFacetDistribTest, TestCloudNestedDocsSort, TestSolrCoreSnapshots, DisMaxRequestHandlerTest, DistributedFacetPivotLongTailTest, TestManagedSchema, HdfsCollectionsAPIDistributedZkTest, TestExportWriter, TestGraphMLResponseWriter, TestBackupRepositoryFactory, TestSSLRandomization, ShardRoutingTest, AlternateDirectoryTest, UnloadDistributedZkTest, TestPullReplica, DirectSolrConnectionTest, TestSolrFieldCacheBean, SortSpecParsingTest, IndexSchemaRuntimeFieldTest, XsltUpdateRequestHandlerTest, TestBlendedInfixSuggestions, HdfsThreadLeakTest, PluginInfoTest, AutoScalingHandlerTest, TestMultiValuedNumericRangeQuery, DocExpirationUpdateProcessorFactoryTest, TestFieldCacheSort, TestHdfsBackupRestoreCore, TestInitQParser, TestSurroundQueryParser, UUIDUpdateProcessorFallbackTest, ConnectionReuseTest, ResponseHeaderTest, TestRemoteStreaming, TestFreeTextSuggestions, TestRandomDVFaceting, DistributedQueryComponentCustomSortTest, TestRTGBase, TestSimpleQParserPlugin, SparseHLLTest, TestCollationField, ChaosMonkeySafeLeaderWithPullReplicasTest, TestRawTransformer, PropertiesRequestHandlerTest, DateRangeFieldTest, TestNamedUpdateProcessors, CoreAdminRequestStatusTest, TestDeleteCollectionOnDownNodes, CdcrBootstrapTest, SynonymTokenizerTest, TestLFUCache, TestUtils, SimpleMLTQParserTest, TestCSVResponseWriter, DirectSolrSpellCheckerTest, TestRuleBasedAuthorizationPlugin, DistributedMLTComponentTest, TestPushWriter, TestConfigSetImmutable, TestLazyCores, CreateCollectionCleanupTest, OpenExchangeRatesOrgProviderTest, TestJsonRequest, HdfsDirectoryFactoryTest, TestSystemIdResolver, TestSolrConfigHandlerCloud, TestStressUserVersions, TestSolrDeletionPolicy1, TestDownShardTolerantSearch, SpellingQueryConverterTest, TestComplexPhraseQParserPlugin, LeaderElectionTest, RecoveryAfterSoftCommitTest, TestIBSimilarityFactory, LeaderElectionContextKeyTest, TestRTimerTree, TestFieldCacheVsDocValues, ClassificationUpdateProcessorFactoryTest, HdfsBasicDistributedZk2Test, TestCSVLoader, TestInPlaceUpdatesStandalone, TestLegacyNumericUtils, TestShortCircuitedRequests, OverseerCollectionConfigSetProcessorTest, SolrSlf4jReporterTest, IndexBasedSpellCheckerTest, HdfsLockFactoryTest, TestQuerySenderNoQuery, LeaderInitiatedRecoveryOnShardRestartTest, TestSchemaSimilarityResource, TestFastOutputStream, TestSolrJ, MBeansHandlerTest, TestNumericTerms64, TestJsonFacetRefinement, SOLR749Test, TestDelegationWithHadoopAuth, TestTolerantUpdateProcessorRandomCloud, TestPerFieldSimilarity, TestFieldSortValues, BlockJoinFacetSimpleTest, TestHashPartitioner, TestGroupingSearch, DeleteInactiveReplicaTest, FastVectorHighlighterTest, TestCloudDeleteByQuery, FileUtilsTest, AddBlockUpdateTest, HighlighterMaxOffsetTest, TestPseudoReturnFields, ConfigureRecoveryStrategyTest, PingRequestHandlerTest, StatelessScriptUpdateProcessorFactoryTest, TestPayloadScoreQParserPlugin, DistributedSuggestComponentTest, TestInitParams, BasicAuthStandaloneTest, TestReRankQParserPlugin, TestConfigSetsAPIZkFailure, QueryEqualityTest, TestGraphTermsQParserPlugin, TestScoreJoinQPNoScore, CollectionTooManyReplicasTest, TestUseDocValuesAsStored, CleanupOldIndexTest, TestImplicitCoreProperties, DistribCursorPagingTest, HdfsUnloadDistributedZkTest, BasicDistributedZkTest, OverseerTest, RecoveryZkTest, TestReplicationHandler, TestDistributedSearch, DistributedSpellCheckComponentTest, ZkControllerTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkControllerTest -Dtests.seed=CF3593D6485D48D -Dtests.slow=true -Dtests.locale=hi -Dtests.timezone=Asia/Dushanbe -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | ZkControllerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [Overseer]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer
   [junit4]    > at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > at org.apache.solr.cloud.Overseer.start(Overseer.java:527)
   [junit4]    > at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:774)
   [junit4]    > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
   [junit4]    > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
   [junit4]    > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
   [junit4]    > at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:393)
   [junit4]    > at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1777)
   [junit4]    > at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([CF3593D6485D48D]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:295)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ZkControllerTest:
   [junit4]    >    1) Thread[id=16555, name=OverseerStateUpdate-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer state updater.]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
   [junit4]    >         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)
   [junit4]    >         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    2) Thread[id=16557, name=OverseerHdfsCoreFailoverThread-98157001021259779-127.0.0.1:8983_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([CF3593D6485D48D]:0)
   [junit4] Completed [389/725 (1!)] on J0 in 43.35s, 4 tests, 1 failure, 1 error, 1 skipped <<< FAILURES!

[...truncated 44494 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4084 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4084/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([6B5CAC29179DE020:3E0C44BBBB642FD0]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1153)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:836)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([6B5CAC29179DE020:9315268D813801E]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.junit.Assert.assertNotNull(Assert.java:537)
        at org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter(MetricsHandlerTest.java:201)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12494 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/init-core-data-001
   [junit4]   2> 1802616 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1802622 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1802623 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001
   [junit4]   2> 1802623 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1802624 INFO  (Thread-4074) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1802624 INFO  (Thread-4074) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1802629 ERROR (Thread-4074) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1802734 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64256
   [junit4]   2> 1802773 INFO  (jetty-launcher-4242-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1802773 INFO  (jetty-launcher-4242-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1802773 INFO  (jetty-launcher-4242-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1802773 INFO  (jetty-launcher-4242-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1802777 INFO  (jetty-launcher-4242-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@716f704f{/solr,null,AVAILABLE}
   [junit4]   2> 1802778 INFO  (jetty-launcher-4242-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@532946a7{/solr,null,AVAILABLE}
   [junit4]   2> 1802778 INFO  (jetty-launcher-4242-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@354062b6{HTTP/1.1,[http/1.1]}{127.0.0.1:64258}
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-1) [    ] o.e.j.s.Server Started @1807413ms
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64258}
   [junit4]   2> 1802779 ERROR (jetty-launcher-4242-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d13a911{/solr,null,AVAILABLE}
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T11:48:45.835Z
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@203fb405{/solr,null,AVAILABLE}
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3b2ad2ae{HTTP/1.1,[http/1.1]}{127.0.0.1:64260}
   [junit4]   2> 1802779 INFO  (jetty-launcher-4242-thread-4) [    ] o.e.j.s.Server Started @1807414ms
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64260}
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@78e4c79b{HTTP/1.1,[http/1.1]}{127.0.0.1:64261}
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.e.j.s.Server Started @1807414ms
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64261}
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@527a5b98{HTTP/1.1,[http/1.1]}{127.0.0.1:64259}
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-3) [    ] o.e.j.s.Server Started @1807414ms
   [junit4]   2> 1802780 ERROR (jetty-launcher-4242-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1802780 ERROR (jetty-launcher-4242-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64259}
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1802780 ERROR (jetty-launcher-4242-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1802780 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T11:48:45.836Z
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T11:48:45.837Z
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1802781 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T11:48:45.837Z
   [junit4]   2> 1802785 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1802785 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1802785 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1802786 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1802796 INFO  (jetty-launcher-4242-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64256/solr
   [junit4]   2> 1802796 INFO  (jetty-launcher-4242-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64256/solr
   [junit4]   2> 1802796 INFO  (jetty-launcher-4242-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64256/solr
   [junit4]   2> 1802798 INFO  (jetty-launcher-4242-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64256/solr
   [junit4]   2> 1802832 INFO  (jetty-launcher-4242-thread-1) [n:127.0.0.1:64258_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1802832 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1802832 INFO  (jetty-launcher-4242-thread-3) [n:127.0.0.1:64259_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1802832 INFO  (jetty-launcher-4242-thread-4) [n:127.0.0.1:64260_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1802833 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64261_solr
   [junit4]   2> 1802834 INFO  (jetty-launcher-4242-thread-1) [n:127.0.0.1:64258_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64258_solr
   [junit4]   2> 1802834 INFO  (jetty-launcher-4242-thread-3) [n:127.0.0.1:64259_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64259_solr
   [junit4]   2> 1802834 INFO  (jetty-launcher-4242-thread-4) [n:127.0.0.1:64260_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64260_solr
   [junit4]   2> 1802835 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.c.Overseer Overseer (id=98158937747226633-127.0.0.1:64261_solr-n_0000000000) starting
   [junit4]   2> 1802836 INFO  (zkCallback-4264-thread-1-processing-n:127.0.0.1:64260_solr) [n:127.0.0.1:64260_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1802836 INFO  (zkCallback-4261-thread-1-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1802836 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1802836 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1802908 INFO  (jetty-launcher-4242-thread-4) [n:127.0.0.1:64260_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802911 INFO  (jetty-launcher-4242-thread-4) [n:127.0.0.1:64260_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802911 INFO  (jetty-launcher-4242-thread-4) [n:127.0.0.1:64260_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802912 INFO  (jetty-launcher-4242-thread-4) [n:127.0.0.1:64260_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/.
   [junit4]   2> 1802913 INFO  (jetty-launcher-4242-thread-1) [n:127.0.0.1:64258_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64258.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802917 INFO  (jetty-launcher-4242-thread-1) [n:127.0.0.1:64258_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64258.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802917 INFO  (jetty-launcher-4242-thread-1) [n:127.0.0.1:64258_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64258.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802917 INFO  (jetty-launcher-4242-thread-1) [n:127.0.0.1:64258_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/.
   [junit4]   2> 1802943 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64261_solr
   [junit4]   2> 1802946 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1802946 INFO  (zkCallback-4261-thread-1-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1802955 INFO  (zkCallback-4264-thread-1-processing-n:127.0.0.1:64260_solr) [n:127.0.0.1:64260_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1802955 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1802965 INFO  (jetty-launcher-4242-thread-3) [n:127.0.0.1:64259_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64259.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802968 INFO  (jetty-launcher-4242-thread-3) [n:127.0.0.1:64259_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64259.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802968 INFO  (jetty-launcher-4242-thread-3) [n:127.0.0.1:64259_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64259.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1802969 INFO  (jetty-launcher-4242-thread-3) [n:127.0.0.1:64259_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node3/.
   [junit4]   2> 1802999 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64261.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1803001 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64261.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1803002 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64261.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1803002 INFO  (jetty-launcher-4242-thread-2) [n:127.0.0.1:64261_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node2/.
   [junit4]   2> 1803025 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1803027 INFO  (SUITE-CustomCollectionTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64256/solr ready
   [junit4]   2> 1803051 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateShardRepFactor
   [junit4]   2> 1803053 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[6B5CAC29179DE020]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1803057 INFO  (qtp1995645580-20568) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1803059 INFO  (OverseerThreadFactory-9280-thread-1-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCreateShardRepFactor
   [junit4]   2> 1803181 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1803181 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1803182 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1803182 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1803301 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1803301 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1804210 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1804210 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1804231 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.IndexSchema [testCreateShardRepFactor_a_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1804232 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.IndexSchema [testCreateShardRepFactor_b_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1804248 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1804248 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_a_replica_n1' using configuration from collection testCreateShardRepFactor, trusted=true
   [junit4]   2> 1804248 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1804248 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_b_replica_n1' using configuration from collection testCreateShardRepFactor, trusted=true
   [junit4]   2> 1804248 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64259.solr.core.testCreateShardRepFactor.a.replica_n1' (registry 'solr.core.testCreateShardRepFactor.a.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1804248 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1804248 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrCore [[testCreateShardRepFactor_a_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node3/testCreateShardRepFactor_a_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node3/./testCreateShardRepFactor_a_replica_n1/data/]
   [junit4]   2> 1804248 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64258.solr.core.testCreateShardRepFactor.b.replica_n1' (registry 'solr.core.testCreateShardRepFactor.b.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1804249 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1804249 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrCore [[testCreateShardRepFactor_b_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/testCreateShardRepFactor_b_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/./testCreateShardRepFactor_b_replica_n1/data/]
   [junit4]   2> 1804300 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1804300 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1804300 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1804300 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1804302 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1804302 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1804302 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1804302 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1804303 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@85a2175[testCreateShardRepFactor_b_replica_n1] main]
   [junit4]   2> 1804303 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@453340d1[testCreateShardRepFactor_a_replica_n1] main]
   [junit4]   2> 1804304 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1804304 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1804305 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1804305 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1804305 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1804305 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1804305 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570543005714087936
   [junit4]   2> 1804306 INFO  (searcherExecutor-9285-thread-1-processing-n:127.0.0.1:64259_solr x:testCreateShardRepFactor_a_replica_n1 s:a c:testCreateShardRepFactor) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica_n1] Registered new searcher Searcher@453340d1[testCreateShardRepFactor_a_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1804306 INFO  (searcherExecutor-9286-thread-1-processing-n:127.0.0.1:64258_solr x:testCreateShardRepFactor_b_replica_n1 s:b c:testCreateShardRepFactor) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica_n1] Registered new searcher Searcher@85a2175[testCreateShardRepFactor_b_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1804305 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570543005714087936
   [junit4]   2> 1804313 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1804313 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1804313 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1804313 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64259/solr/testCreateShardRepFactor_a_replica_n1/
   [junit4]   2> 1804313 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1804313 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1804313 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64258/solr/testCreateShardRepFactor_b_replica_n1/
   [junit4]   2> 1804313 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:64259/solr/testCreateShardRepFactor_a_replica_n1/ has no replicas
   [junit4]   2> 1804313 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1804313 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:64258/solr/testCreateShardRepFactor_b_replica_n1/ has no replicas
   [junit4]   2> 1804313 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1804313 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1804318 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64258/solr/testCreateShardRepFactor_b_replica_n1/ b
   [junit4]   2> 1804318 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64259/solr/testCreateShardRepFactor_a_replica_n1/ a
   [junit4]   2> 1804429 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1804429 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1804487 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1804487 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1804493 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1312
   [junit4]   2> 1804495 INFO  (qtp1802555980-20555) [n:127.0.0.1:64258_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica_n1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1313
   [junit4]   2> 1804502 INFO  (qtp1995645580-20568) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1804613 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1804613 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1805068 INFO  (OverseerCollectionConfigSetProcessor-98158937747226633-127.0.0.1:64261_solr-n_0000000000) [n:127.0.0.1:64261_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1805508 INFO  (qtp1995645580-20568) [n:127.0.0.1:64260_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2} status=0 QTime=2451
   [junit4]   2> 1805509 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createshard with params action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1805515 INFO  (OverseerThreadFactory-9280-thread-2-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.CreateShardCmd Create shard invoked: {
   [junit4]   2>   "collection":"testCreateShardRepFactor",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "operation":"createshard"}
   [junit4]   2> 1805634 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1805634 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1805740 INFO  (OverseerThreadFactory-9280-thread-2-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.CreateShardCmd Creating replica testCreateShardRepFactor_x_replica_n1 as part of slice x of collection testCreateShardRepFactor on 127.0.0.1:64260_solr
   [junit4]   2> 1805743 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&name=testCreateShardRepFactor_x_replica_n1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1805745 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1805862 INFO  (zkCallback-4264-thread-1-processing-n:127.0.0.1:64260_solr) [n:127.0.0.1:64260_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1805862 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1805862 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1806783 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1806795 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.IndexSchema [testCreateShardRepFactor_x_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1806820 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1806820 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_x_replica_n1' using configuration from collection testCreateShardRepFactor, trusted=true
   [junit4]   2> 1806821 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.core.testCreateShardRepFactor.x.replica_n1' (registry 'solr.core.testCreateShardRepFactor.x.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1806821 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1806821 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrCore [[testCreateShardRepFactor_x_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/testCreateShardRepFactor_x_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/./testCreateShardRepFactor_x_replica_n1/data/]
   [junit4]   2> 1806868 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1806868 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1806869 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1806869 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1806870 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@389eccea[testCreateShardRepFactor_x_replica_n1] main]
   [junit4]   2> 1806872 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1806872 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1806872 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1806873 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570543008406831104
   [junit4]   2> 1806873 INFO  (searcherExecutor-9295-thread-1-processing-n:127.0.0.1:64260_solr x:testCreateShardRepFactor_x_replica_n1 s:x c:testCreateShardRepFactor) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica_n1] Registered new searcher Searcher@389eccea[testCreateShardRepFactor_x_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1806879 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1806879 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1806879 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64260/solr/testCreateShardRepFactor_x_replica_n1/
   [junit4]   2> 1806879 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1806879 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:64260/solr/testCreateShardRepFactor_x_replica_n1/ has no replicas
   [junit4]   2> 1806879 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1806883 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64260/solr/testCreateShardRepFactor_x_replica_n1/ x
   [junit4]   2> 1806991 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1806991 INFO  (zkCallback-4264-thread-1-processing-n:127.0.0.1:64260_solr) [n:127.0.0.1:64260_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1806991 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1806994 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1806999 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&name=testCreateShardRepFactor_x_replica_n1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1256
   [junit4]   2> 1807000 INFO  (OverseerThreadFactory-9280-thread-2-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.CreateShardCmd Finished create command on all shards for collection: testCreateShardRepFactor
   [junit4]   2> 1807004 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2} status=0 QTime=1495
   [junit4]   2> 1807007 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateShardRepFactor
   [junit4]   2> 1807054 INFO  (TEST-CustomCollectionTest.testRouteFieldForImplicitRouter-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRouteFieldForImplicitRouter
   [junit4]   2> 1807058 INFO  (TEST-CustomCollectionTest.testRouteFieldForImplicitRouter-seed#[6B5CAC29179DE020]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1807062 INFO  (qtp1995645580-20576) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1807068 INFO  (OverseerCollectionConfigSetProcessor-98158937747226633-127.0.0.1:64261_solr-n_0000000000) [n:127.0.0.1:64261_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1807069 INFO  (OverseerThreadFactory-9280-thread-3-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 1807071 INFO  (qtp1279988014-20567) [n:127.0.0.1:64259_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.a.replica_n1, tag=null
   [junit4]   2> 1807071 INFO  (qtp1802555980-20557) [n:127.0.0.1:64258_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.b.replica_n1, tag=null
   [junit4]   2> 1807071 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.x.replica_n1, tag=null
   [junit4]   2> 1807103 INFO  (zkCallback-4264-thread-1-processing-n:127.0.0.1:64260_solr) [n:127.0.0.1:64260_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1807103 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1807103 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1807136 INFO  (qtp1279988014-20567) [n:127.0.0.1:64259_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@17849c7
   [junit4]   2> 1807137 INFO  (qtp1802555980-20557) [n:127.0.0.1:64258_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@bb2fb00
   [junit4]   2> 1807137 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@749b868e
   [junit4]   2> 1807137 INFO  (qtp1279988014-20567) [n:127.0.0.1:64259_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.a.replica_n1, tag=24660423
   [junit4]   2> 1807137 INFO  (qtp1279988014-20567) [n:127.0.0.1:64259_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCreateShardRepFactor.a.leader, tag=24660423
   [junit4]   2> 1807138 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.x.replica_n1, tag=1956349582
   [junit4]   2> 1807138 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCreateShardRepFactor.x.leader, tag=1956349582
   [junit4]   2> 1807138 INFO  (qtp1802555980-20557) [n:127.0.0.1:64258_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCreateShardRepFactor.b.replica_n1, tag=196279040
   [junit4]   2> 1807138 INFO  (qtp1802555980-20557) [n:127.0.0.1:64258_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCreateShardRepFactor.b.leader, tag=196279040
   [junit4]   2> 1807141 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_x_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=70
   [junit4]   2> 1807141 INFO  (qtp1279988014-20567) [n:127.0.0.1:64259_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_a_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=71
   [junit4]   2> 1807142 INFO  (qtp1802555980-20557) [n:127.0.0.1:64258_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_b_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=71
   [junit4]   2> 1807876 INFO  (qtp1995645580-20576) [n:127.0.0.1:64260_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2} status=0 QTime=814
   [junit4]   2> 1807878 INFO  (qtp1995645580-20578) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b,c,d&replicationFactor=5&collection.configName=conf&maxShardsPerNode=6&router.field=shard_s&name=withShardField&router.name=implicit&nrtReplicas=5&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1807883 INFO  (OverseerThreadFactory-9280-thread-4-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.CreateCollectionCmd Create collection withShardField
   [junit4]   2> 1807884 WARN  (OverseerThreadFactory-9280-thread-4-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 5 on collection withShardField is higher than the number of Solr instances currently live or live and part of your createNodeSet(4). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 1807884 INFO  (OverseerCollectionConfigSetProcessor-98158937747226633-127.0.0.1:64261_solr-n_0000000000) [n:127.0.0.1:64261_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1808107 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica_n3&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808108 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica_n2&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808110 INFO  (qtp1995645580-20568) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica_n5&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808114 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica_n1&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808122 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica_n4&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808123 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica_n3&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808126 INFO  (qtp1279988014-20569) [n:127.0.0.1:64259_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica_n5&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808131 INFO  (qtp1279988014-20571) [n:127.0.0.1:64259_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica_n3&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808131 INFO  (qtp1279988014-20573) [n:127.0.0.1:64259_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica_n1&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808134 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica_n4&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808134 INFO  (qtp1802555980-20560) [n:127.0.0.1:64258_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica_n2&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808134 INFO  (qtp445945835-20581) [n:127.0.0.1:64261_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica_n3&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808135 INFO  (qtp445945835-20581) [n:127.0.0.1:64261_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1808135 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica_n5&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808135 INFO  (qtp1802555980-20561) [n:127.0.0.1:64258_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica_n5&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808135 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica_n1&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808136 INFO  (qtp1802555980-20553) [n:127.0.0.1:64258_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica_n1&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808136 INFO  (qtp445945835-20586) [n:127.0.0.1:64261_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica_n4&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808137 INFO  (qtp445945835-20587) [n:127.0.0.1:64261_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica_n2&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808138 INFO  (qtp1279988014-20556) [n:127.0.0.1:64259_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica_n2&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808138 INFO  (qtp1279988014-20565) [n:127.0.0.1:64259_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica_n4&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1808278 INFO  (zkCallback-4264-thread-1-processing-n:127.0.0.1:64260_solr) [n:127.0.0.1:64260_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1808279 INFO  (zkCallback-4263-thread-1-processing-n:127.0.0.1:64258_solr) [n:127.0.0.1:64258_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1808279 INFO  (zkCallback-4261-thread-1-processing-n:127.0.0.1:64261_solr) [n:127.0.0.1:64261_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1808278 INFO  (zkCallback-4262-thread-1-processing-n:127.0.0.1:64259_solr) [n:127.0.0.1:64259_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1809158 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809159 INFO  (qtp1995645580-20568) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809159 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809196 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809198 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.s.IndexSchema [withShardField_c_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 1809201 INFO  (qtp1995645580-20568) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n5] o.a.s.s.IndexSchema [withShardField_d_replica_n5] Schema name=cloud-dynamic
   [junit4]   2> 1809202 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.s.IndexSchema [withShardField_c_replica_n3] Schema name=cloud-dynamic
   [junit4]   2> 1809208 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809212 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr c:withShardField s:b  x:withShardField_b_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809218 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.s.IndexSchema [withShardField_d_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1809219 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809219 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_c_replica_n3' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809219 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64258.solr.core.withShardField.c.replica_n3' (registry 'solr.core.withShardField.c.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809219 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809219 INFO  (qtp1802555980-20558) [n:127.0.0.1:64258_solr c:withShardField s:c  x:withShardField_c_replica_n3] o.a.s.c.SolrCore [[withShardField_c_replica_n3] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/withShardField_c_replica_n3], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/./withShardField_c_replica_n3/data/]
   [junit4]   2> 1809222 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.s.IndexSchema [withShardField_a_replica_n4] Schema name=cloud-dynamic
   [junit4]   2> 1809245 INFO  (qtp1279988014-20569) [n:127.0.0.1:64259_solr c:withShardField s:c  x:withShardField_c_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809247 INFO  (qtp445945835-20586) [n:127.0.0.1:64261_solr c:withShardField s:c  x:withShardField_c_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809249 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809250 INFO  (qtp445945835-20587) [n:127.0.0.1:64261_solr c:withShardField s:a  x:withShardField_a_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809253 INFO  (qtp445945835-20581) [n:127.0.0.1:64261_solr c:withShardField s:d  x:withShardField_d_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809255 INFO  (qtp1802555980-20560) [n:127.0.0.1:64258_solr c:withShardField s:d  x:withShardField_d_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809255 INFO  (qtp1802555980-20561) [n:127.0.0.1:64258_solr c:withShardField s:a  x:withShardField_a_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809258 INFO  (qtp1802555980-20553) [n:127.0.0.1:64258_solr c:withShardField s:a  x:withShardField_a_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809261 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809262 INFO  (qtp445945835-20586) [n:127.0.0.1:64261_solr c:withShardField s:c  x:withShardField_c_replica_n4] o.a.s.s.IndexSchema [withShardField_c_replica_n4] Schema name=cloud-dynamic
   [junit4]   2> 1809264 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.s.IndexSchema [withShardField_b_replica_n5] Schema name=cloud-dynamic
   [junit4]   2> 1809290 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809290 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_d_replica_n1' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809291 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.core.withShardField.d.replica_n1' (registry 'solr.core.withShardField.d.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809291 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809291 INFO  (qtp1995645580-20575) [n:127.0.0.1:64260_solr c:withShardField s:d  x:withShardField_d_replica_n1] o.a.s.c.SolrCore [[withShardField_d_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/withShardField_d_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/./withShardField_d_replica_n1/data/]
   [junit4]   2> 1809292 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809292 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_b_replica_n5' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809292 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64261.solr.core.withShardField.b.replica_n5' (registry 'solr.core.withShardField.b.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809292 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809292 INFO  (qtp445945835-20584) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n5] o.a.s.c.SolrCore [[withShardField_b_replica_n5] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node2/withShardField_b_replica_n5], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node2/./withShardField_b_replica_n5/data/]
   [junit4]   2> 1809294 INFO  (qtp445945835-20587) [n:127.0.0.1:64261_solr c:withShardField s:a  x:withShardField_a_replica_n2] o.a.s.s.IndexSchema [withShardField_a_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 1809294 INFO  (qtp1802555980-20560) [n:127.0.0.1:64258_solr c:withShardField s:d  x:withShardField_d_replica_n2] o.a.s.s.IndexSchema [withShardField_d_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 1809295 INFO  (qtp445945835-20581) [n:127.0.0.1:64261_solr c:withShardField s:d  x:withShardField_d_replica_n3] o.a.s.s.IndexSchema [withShardField_d_replica_n3] Schema name=cloud-dynamic
   [junit4]   2> 1809302 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809303 INFO  (qtp1279988014-20573) [n:127.0.0.1:64259_solr c:withShardField s:c  x:withShardField_c_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809307 INFO  (qtp1279988014-20571) [n:127.0.0.1:64259_solr c:withShardField s:a  x:withShardField_a_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809307 INFO  (qtp1279988014-20556) [n:127.0.0.1:64259_solr c:withShardField s:b  x:withShardField_b_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1809309 INFO  (qtp1802555980-20561) [n:127.0.0.1:64258_solr c:withShardField s:a  x:withShardField_a_replica_n5] o.a.s.s.IndexSchema [withShardField_a_replica_n5] Schema name=cloud-dynamic
   [junit4]   2> 1809313 INFO  (qtp1802555980-20553) [n:127.0.0.1:64258_solr c:withShardField s:a  x:withShardField_a_replica_n1] o.a.s.s.IndexSchema [withShardField_a_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1809313 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.s.IndexSchema [withShardField_b_replica_n4] Schema name=cloud-dynamic
   [junit4]   2> 1809328 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.s.IndexSchema [withShardField_b_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1809339 INFO  (qtp1995645580-20577) [n:127.0.0.1:64260_solr c:withShardField s:b  x:withShardField_b_replica_n3] o.a.s.s.IndexSchema [withShardField_b_replica_n3] Schema name=cloud-dynamic
   [junit4]   2> 1809341 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809341 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_b_replica_n4' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809342 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64258.solr.core.withShardField.b.replica_n4' (registry 'solr.core.withShardField.b.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809342 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809342 INFO  (qtp1802555980-20559) [n:127.0.0.1:64258_solr c:withShardField s:b  x:withShardField_b_replica_n4] o.a.s.c.SolrCore [[withShardField_b_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/withShardField_b_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node1/./withShardField_b_replica_n4/data/]
   [junit4]   2> 1809344 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809344 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_b_replica_n1' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809345 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64261.solr.core.withShardField.b.replica_n1' (registry 'solr.core.withShardField.b.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809345 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809345 INFO  (qtp445945835-20585) [n:127.0.0.1:64261_solr c:withShardField s:b  x:withShardField_b_replica_n1] o.a.s.c.SolrCore [[withShardField_b_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node2/withShardField_b_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node2/./withShardField_b_replica_n1/data/]
   [junit4]   2> 1809350 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809350 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_a_replica_n4' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809350 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.core.withShardField.a.replica_n4' (registry 'solr.core.withShardField.a.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809350 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809350 INFO  (qtp1995645580-20574) [n:127.0.0.1:64260_solr c:withShardField s:a  x:withShardField_a_replica_n4] o.a.s.c.SolrCore [[withShardField_a_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/withShardField_a_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/./withShardField_a_replica_n4/data/]
   [junit4]   2> 1809351 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1809351 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_c_replica_n2' using configuration from collection withShardField, trusted=true
   [junit4]   2> 1809351 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64260.solr.core.withShardField.c.replica_n2' (registry 'solr.core.withShardField.c.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4338516a
   [junit4]   2> 1809351 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1809351 INFO  (qtp1995645580-20562) [n:127.0.0.1:64260_solr c:withShardField s:c  x:withShardField_c_replica_n2] o.a.s.c.SolrCore [[withShardField_c_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/withShardField_c_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_6B5CAC29179DE020-001/tempDir-001/node4/./withShardField_c_replica_n2/data/]
   [junit4]   2> 1809358 INFO  (qtp1279988014-20556) [n:127.0.0.1:64259_solr c:withShardField s:b  x:withShardField_b_replica_n2] o.a.s.s.IndexSchema [withShardField_b_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 1809358 INFO  (qtp1279988014-20573) [n:127.0.0.1:64259_solr c:withShardField s:c  x:withShardField_c_replica_n1] o.a.s.s.IndexSchema [withShardField_c_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1

[...truncated too long message...]

ion1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3ec1719[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2412153 INFO  (TEST-MetricsHandlerTest.test-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2412159 INFO  (TEST-MetricsHandlerTest.test-seed#[6B5CAC29179DE020]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2412202 INFO  (TEST-MetricsHandlerTest.test-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> 2412205 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCompact
   [junit4]   2> 2412213 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[6B5CAC29179DE020]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCompact
   [junit4]   2> 2412213 INFO  (SUITE-MetricsHandlerTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 2412213 INFO  (SUITE-MetricsHandlerTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=567127269
   [junit4]   2> 2412213 INFO  (SUITE-MetricsHandlerTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2412214 INFO  (SUITE-MetricsHandlerTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2412215 INFO  (SUITE-MetricsHandlerTest-seed#[6B5CAC29179DE020]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2412215 INFO  (coreCloseExecutor-11698-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@45082c8
   [junit4]   2> 2412221 INFO  (coreCloseExecutor-11698-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=72385224
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.admin.MetricsHandlerTest_6B5CAC29179DE020-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=241, maxMBSortInHeap=7.625335995488003, sim=RandomSimilarity(queryNorm=false): {}, locale=ja-JP, timezone=US/Michigan
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=96363144,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestLRUStatsCache, TestTestInjection, BufferStoreTest, AnalyticsMergeStrategyTest, ConfigSetsAPITest, DateFieldTest, TestTolerantSearch, StandardRequestHandlerTest, TestFieldCollectionResource, TestEmbeddedSolrServerSchemaAPI, SortSpecParsingTest, SolrCLIZkUtilsTest, CollectionsAPISolrJTest, TestApiFramework, TestJsonRequest, RulesTest, TestFieldTypeCollectionResource, ParsingFieldUpdateProcessorsTest, DistributedFacetPivotLongTailTest, DistributedDebugComponentTest, TestDownShardTolerantSearch, TestStressReorder, PingRequestHandlerTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestBulkSchemaConcurrent, TestTlogReplica, TestSimpleTextCodec, LeaderElectionContextKeyTest, DistributedFacetPivotSmallTest, ShardRoutingTest, TestExtendedDismaxParser, TestPartialUpdateDeduplication, DistributedMLTComponentTest, TestSystemIdResolver, TestSimpleQParserPlugin, XmlUpdateRequestHandlerTest, EnumFieldTest, SuggesterFSTTest, TestSerializedLuceneMatchVersion, SearchHandlerTest, TestHashPartitioner, SimpleMLTQParserTest, StatelessScriptUpdateProcessorFactoryTest, SolrMetricsIntegrationTest, TestCSVLoader, TestOverriddenPrefixQueryForCustomFieldType, CdcrVersionReplicationTest, SSLMigrationTest, TestLocalFSCloudBackupRestore, TestChildDocTransformer, SparseHLLTest, SchemaVersionSpecificBehaviorTest, TestManagedSynonymGraphFilterFactory, TestSolrXml, TestLFUCache, SoftAutoCommitTest, TestFieldCache, DirectSolrConnectionTest, TestSortByMinMaxFunction, TestRandomDVFaceting, DeleteNodeTest, UUIDUpdateProcessorFallbackTest, TestRawTransformer, AliasIntegrationTest, TestReloadDeadlock, TestMaxTokenLenTokenizer, TestConfigSetImmutable, ZkCLITest, TestSchemaManager, ClusterStateUpdateTest, SharedFSAutoReplicaFailoverUtilsTest, TestPostingsSolrHighlighter, TestSQLHandler, TestSweetSpotSimilarityFactory, DisMaxRequestHandlerTest, TestExactSharedStatsCache, TestFieldCacheSort, TestReplicationHandler, DistributedSuggestComponentTest, TestMultiValuedNumericRangeQuery, SpatialFilterTest, OverseerCollectionConfigSetProcessorTest, TestBlendedInfixSuggestions, SimplePostToolTest, SolrPluginUtilsTest, DirectoryFactoryTest, IndexSchemaRuntimeFieldTest, TestPayloadScoreQParserPlugin, TestRandomCollapseQParserPlugin, TestFreeTextSuggestions, TestHttpShardHandlerFactory, DocumentAnalysisRequestHandlerTest, HdfsChaosMonkeyNothingIsSafeTest, PropertiesRequestHandlerTest, TestSolrCoreSnapshots, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, DocExpirationUpdateProcessorFactoryTest, DistributedVersionInfoTest, LeaderInitiatedRecoveryOnShardRestartTest, SuggestComponentContextFilterQueryTest, TestMergePolicyConfig, TestRuleBasedAuthorizationPlugin, TestLegacyNumericUtils, TestCloudDeleteByQuery, V2StandaloneTest, TestSolrFieldCacheBean, TestSolrConfigHandlerCloud, PluginInfoTest, WrapperMergePolicyFactoryTest, TestInPlaceUpdatesStandalone, HdfsLockFactoryTest, ConfigureRecoveryStrategyTest, TestReqParamsAPI, TestSort, CdcrUpdateLogTest, TestConfigSets, SyncSliceTest, DocValuesTest, TestSurroundQueryParser, TestPseudoReturnFields, TestMaxScoreQueryParser, IndexBasedSpellCheckerTest, TestNumericRangeQuery64, SimpleFacetsTest, TestNamedUpdateProcessors, InfoHandlerTest, MergeStrategyTest, TestRestManager, TestCustomStream, TestDelegationWithHadoopAuth, TestLegacyField, TestFieldSortValues, ClassificationUpdateProcessorTest, CollectionTooManyReplicasTest, DistributedQueryElevationComponentTest, TestManagedSchemaThreadSafety, DistribJoinFromCollectionTest, OpenExchangeRatesOrgProviderTest, InfixSuggestersTest, ForceLeaderTest, OverseerTaskQueueTest, TestLeaderElectionZkExpiry, TestQuerySenderNoQuery, DistributedFacetExistsSmallTest, CdcrBootstrapTest, OverseerRolesTest, TestCharFilters, TestHdfsBackupRestoreCore, TestCollapseQParserPlugin, TestCloudPseudoReturnFields, CollectionReloadTest, TestInitParams, CoreAdminRequestStatusTest, HighlighterTest, TestPushWriter, TestReRankQParserPlugin, TestClassNameShortening, TestFastOutputStream, TestSolrDeletionPolicy1, TestRandomFlRTGCloud, TestSubQueryTransformerCrossCore, OverseerTest, TestRebalanceLeaders, DocumentBuilderTest, TestAuthorizationFramework, TestShortCircuitedRequests, TestDFISimilarityFactory, XsltUpdateRequestHandlerTest, CursorPagingTest, CopyFieldTest, CurrencyFieldXmlFileTest, TestFieldCacheVsDocValues, SolrTestCaseJ4Test, TestConfigSetsAPIZkFailure, TestStressInPlaceUpdates, BasicDistributedZkTest, UnloadDistributedZkTest, RecoveryZkTest, ShardRoutingCustomTest, TestDistributedSearch, TestFaceting, DistributedSpellCheckComponentTest, HardAutoCommitTest, TestSolr4Spatial, StatsComponentTest, QueryElevationComponentTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, TestFunctionQuery, TestLazyCores, CoreAdminHandlerTest, SuggesterWFSTTest, TestUpdate, FieldMutatingUpdateProcessorTest, QueryEqualityTest, DocValuesMultiTest, CacheHeaderTest, TestWriterPerf, TestQueryTypes, TermsComponentTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, RequiredFieldsTest, FastVectorHighlighterTest, TestJmxIntegration, TestCSVResponseWriter, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, HighlighterConfigTest, TestQuerySenderListener, UpdateParamsTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, MultiTermTest, SampleTest, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, OutputWriterTest, TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, DistributedIntervalFacetingTest, TestSimpleTrackingShardHandler, CdcrReplicationHandlerTest, CdcrRequestHandlerTest, ChaosMonkeySafeLeaderWithPullReplicasTest, ConcurrentDeleteAndCreateCollectionTest, ConnectionManagerTest, CreateCollectionCleanupTest, CustomCollectionTest, DistribCursorPagingTest, DocValuesNotIndexedTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnCommitTest, MissingSegmentRecoveryTest, NodeMutatorTest, OverseerModifyCollectionTest, RecoveryAfterSoftCommitTest, ReplaceNodeTest, SaslZkACLProviderTest, ShardSplitTest, TestMiniSolrCloudClusterSSL, TestPullReplicaErrorHandling, TestRandomRequestDistribution, TestRequestForwarding, TestSegmentSorting, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation, TestStressLiveNodes, TestTolerantUpdateProcessorCloud, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, HdfsRecoverLeaseTest, HdfsSyncSliceTest, HdfsUnloadDistributedZkTest, StressHdfsTest, ImplicitSnitchTest, RuleEngineTest, HdfsDirectoryFactoryTest, QueryResultKeyTest, TestBackupRepositoryFactory, TestConfigSetProperties, TestImplicitCoreProperties, TestNRTOpen, TestSolrCloudSnapshots, TestSQLHandlerNonCloud, TestSolrConfigHandlerConcurrent, V2ApiIntegrationTest, CoreMergeIndexesAdminHandlerTest, MetricsHandlerTest]
   [junit4] Completed [589/725 (2!)] on J0 in 0.50s, 3 tests, 1 failure <<< FAILURES!

[...truncated 43860 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4085 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4085/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

10 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI

Error Message:
Could not find collection : implicitcoll

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : implicitcoll
        at __randomizedtesting.SeedInfo.seed([B738AE59E6F9646A:DDD92032DB63D212]:0)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:194)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
        at org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
org/apache/solr/client/solrj/request/CollectionAdminRequest$SplitShard

Stack Trace:
java.lang.NoClassDefFoundError: org/apache/solr/client/solrj/request/CollectionAdminRequest$SplitShard
        at __randomizedtesting.SeedInfo.seed([B738AE59E6F9646A:3F6C918348050992]:0)
        at org.apache.solr.client.solrj.request.CollectionAdminRequest.splitShard(CollectionAdminRequest.java:1009)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testDeduplicationOfSubmittedTasks(MultiThreadedOCPTest.java:211)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.client.solrj.request.CollectionAdminRequest$SplitShard
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 44 more
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/classes/java/org/apache/solr/client/solrj/request/CollectionAdminRequest$SplitShard.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 50 more


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
Too many open files

Stack Trace:
java.io.IOException: Too many open files
        at __randomizedtesting.SeedInfo.seed([B738AE59E6F9646A:4E753DF6DA8C29E0]:0)
        at sun.nio.ch.KQueueArrayWrapper.init(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.<init>(KQueueArrayWrapper.java:98)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:88)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.eclipse.jetty.io.ManagedSelector.newSelector(ManagedSelector.java:96)
        at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:90)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:273)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:268)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.server.Server.doStart(Server.java:401)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:558)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:432)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:347)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:983)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitStaticIndexReplication

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([B738AE59E6F9646A:FD723A5977514B6F]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:171)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:107)
        at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:234)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:140)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:397)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:279)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:345)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:983)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1842)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:146)
        ... 40 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1839)
        ... 46 more


FAILED:  org.apache.solr.cloud.ShardSplitTest.test

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([B738AE59E6F9646A:3F6C918348050992]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:171)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:90)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:80)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:80)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:225)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1842)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:146)
        ... 40 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1839)
        ... 46 more


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
Could not get the port for ZooKeeper server

Stack Trace:
java.lang.RuntimeException: Could not get the port for ZooKeeper server
        at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:515)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:70)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:225)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitShardWithRule

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([B738AE59E6F9646A:36367FEB7B0B3F41]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:171)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:107)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:86)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:80)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:80)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:225)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1842)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:146)
        ... 39 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1839)
        ... 45 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest

Error Message:
49 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:     1) Thread[id=16568, name=org.eclipse.jetty.server.session.HashSessionManager@3d0d4d84Timer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=16565, name=qtp1746331879-16565, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=16628, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)    4) Thread[id=16534, name=org.eclipse.jetty.server.session.HashSessionManager@5ce0171cTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=16556, name=org.eclipse.jetty.server.session.HashSessionManager@7508515bTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=16553, name=qtp1413570838-16553, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=16532, name=qtp1658918683-16532, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=16566, name=qtp1746331879-16566, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=16627, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=16533, name=qtp1658918683-16533, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=16621, name=ProcessThread(sid:0 cport:59710):, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)   12) Thread[id=16626, name=Thread-3184, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)   13) Thread[id=16548, name=qtp1413570838-16548, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=16531, name=qtp1658918683-16531, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=16538, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=16608, name=OverseerStateUpdate-98160332505415684-127.0.0.1:59706_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)         at java.lang.Thread.run(Thread.java:748)   17) Thread[id=16530, name=qtp1658918683-16530, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=16544, name=TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[B738AE59E6F9646A]-EventThread, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   19) Thread[id=16555, name=qtp1413570838-16555, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=16528, name=qtp1658918683-16528, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   21) Thread[id=16529, name=qtp1658918683-16529, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   22) Thread[id=16560, name=qtp1746331879-16560, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   23) Thread[id=16547, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   24) Thread[id=16545, name=zkCallback-3347-thread-1, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)   25) Thread[id=16609, name=OverseerCollectionConfigSetProcessor-98160332505415684-127.0.0.1:59706_-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:387)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193)         at java.lang.Thread.run(Thread.java:748)   26) Thread[id=16554, name=qtp1413570838-16554, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   27) Thread[id=16559, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   28) Thread[id=16526, name=qtp1658918683-16526, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   29) Thread[id=16567, name=qtp1746331879-16567, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   30) Thread[id=16549, name=qtp1413570838-16549-acceptor-०@7999ce78-ServerConnector@2b2f72cc{HTTP/1.1,[http/1.1]}{127.0.0.1:५९५६७}, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   31) Thread[id=16527, name=qtp1658918683-16527-acceptor-०@2b536abc-ServerConnector@4335734e{HTTP/1.1,[http/1.1]}{127.0.0.1:५९५६२}, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   32) Thread[id=16600, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   33) Thread[id=16564, name=qtp1746331879-16564, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   34) Thread[id=16550, name=qtp1413570838-16550, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   35) Thread[id=16601, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   36) Thread[id=16606, name=TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[B738AE59E6F9646A]-EventThread, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   37) Thread[id=16610, name=OverseerHdfsCoreFailoverThread-98160332505415684-127.0.0.1:59706_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)         at java.lang.Thread.run(Thread.java:748)   38) Thread[id=16543, name=TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[B738AE59E6F9646A]-SendThread(127.0.0.1:59549), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)   39) Thread[id=16562, name=qtp1746331879-16562, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   40) Thread[id=16552, name=qtp1413570838-16552, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   41) Thread[id=16539, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   42) Thread[id=16551, name=qtp1413570838-16551, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   43) Thread[id=16619, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)   44) Thread[id=16630, name=ProcessThread(sid:0 cport:59954):, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)   45) Thread[id=16561, name=qtp1746331879-16561, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   46) Thread[id=16605, name=TEST-ShardSplitTest.testSplitStaticIndexReplication-seed#[B738AE59E6F9646A]-SendThread(127.0.0.1:59703), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)   47) Thread[id=16563, name=qtp1746331879-16563, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   48) Thread[id=16617, name=Thread-3182, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)   49) Thread[id=16618, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 49 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
   1) Thread[id=16568, name=org.eclipse.jetty.server.session.HashSessionManager@3d0d4d84Timer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=16565, name=qtp1746331879-16565, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=16628, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   4) Thread[id=16534, name=org.eclipse.jetty.server.session.HashSessionManager@5ce0171cTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=16556, name=org.eclipse.jetty.server.session.HashSessionManager@7508515bTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=16553, name=qtp1413570838-16553, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=16532, name=qtp1658918683-16532, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=16566, name=qtp1746331879-16566, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=16627, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=16533, name=qtp1658918683-16533, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=16621, name=ProcessThread(sid:0 cport:59710):, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
  12) Thread[id=16626, name=Thread-3184, state=WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
  13) Thread[id=16548, name=qtp1413570838-16548, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=16531, name=qtp1658918683-16531, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=16538, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  16) Thread[id=16608, name=OverseerStateUpdate-98160332505415684-127.0.0.1:59706_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
        at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:139)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:184)
        at java.lang.Thread.run(Thread.java:748)
  17) Thread[id=16530, name=qtp1658918683-16530, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  18) Thread[id=16544, name=TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[B738AE59E6F9646A]-EventThread, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
  19) Thread[id=16555, name=qtp1413570838-16555, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  20) Thread[id=16528, name=qtp1658918683-16528, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  21) Thread[id=16529, name=qtp1658918683-16529, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  22) Thread[id=16560, name=qtp1746331879-16560, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  23) Thread[id=16547, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  24) Thread[id=16545, name=zkCallback-3347-thread-1, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
  25) Thread[id=16609, name=OverseerCollectionConfigSetProcessor-98160332505415684-127.0.0.1:59706_-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:387)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193)
        at java.lang.Thread.run(Thread.java:748)
  26) Thread[id=16554, name=qtp1413570838-16554, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  27) Thread[id=16559, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.

[...truncated too long message...]

llback-1576-thread-1-processing-n:127.0.0.1:61535_solr x:collection1_collection_shard2_replica_n1 s:shard2 c:collection1_collection r:core_node2) [n:127.0.0.1:61535_solr c:collection1_collection s:shard2 r:core_node2 x:collection1_collection_shard2_replica_n1] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 220644 INFO  (coreCloseExecutor-651-thread-1) [n:127.0.0.1:61535_solr c:collection1_collection s:shard2 r:core_node2 x:collection1_collection_shard2_replica_n1] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2f46b1cc
   [junit4]   2> 220644 INFO  (coreCloseExecutor-652-thread-1) [n:127.0.0.1:61536_solr c:collection1_collection s:shard1 r:core_node1 x:collection1_collection_shard1_replica_n1] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3bec3b1a
   [junit4]   2> 220646 INFO  (zkCallback-187-thread-1-processing-n:127.0.0.1:61536_solr) [n:127.0.0.1:61536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 220646 INFO  (zkCallback-186-thread-2-processing-n:127.0.0.1:61535_solr) [n:127.0.0.1:61535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 220647 INFO  (coreCloseExecutor-652-thread-1) [n:127.0.0.1:61536_solr c:collection1_collection s:shard1 r:core_node1 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1_collection.shard1.replica_n1, tag=1005337370
   [junit4]   2> 220647 INFO  (coreCloseExecutor-651-thread-1) [n:127.0.0.1:61535_solr c:collection1_collection s:shard2 r:core_node2 x:collection1_collection_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1_collection.shard2.replica_n1, tag=793162188
   [junit4]   2> 220676 INFO  (coreCloseExecutor-652-thread-1) [n:127.0.0.1:61536_solr c:collection1_collection s:shard1 r:core_node1 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard1.leader, tag=1005337370
   [junit4]   2> 220677 INFO  (coreCloseExecutor-651-thread-1) [n:127.0.0.1:61535_solr c:collection1_collection s:shard2 r:core_node2 x:collection1_collection_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard2.leader, tag=793162188
   [junit4]   2> 220679 INFO  (jetty-closer-166-thread-4) [    ] o.a.s.c.Overseer Overseer (id=98160355200139276-127.0.0.1:61536_solr-n_0000000000) closing
   [junit4]   2> 220684 INFO  (OverseerStateUpdate-98160355200139276-127.0.0.1:61536_solr-n_0000000000) [n:127.0.0.1:61536_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:61536_solr
   [junit4]   2> 220685 WARN  (zkCallback-186-thread-2-processing-n:127.0.0.1:61535_solr) [n:127.0.0.1:61535_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 220686 INFO  (jetty-closer-166-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@21cdfb50{/solr,null,UNAVAILABLE}
   [junit4]   2> 220686 INFO  (zkCallback-187-thread-1-processing-n:127.0.0.1:61536_solr) [n:127.0.0.1:61536_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 220687 WARN  (zkCallback-187-thread-1-processing-n:127.0.0.1:61536_solr) [n:127.0.0.1:61536_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 220687 INFO  (jetty-closer-166-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@68fd9cd9{/solr,null,UNAVAILABLE}
   [junit4]   2> 220689 ERROR (SUITE-StreamExpressionTest-seed#[EBB1030E59B136A7]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 220689 INFO  (SUITE-StreamExpressionTest-seed#[EBB1030E59B136A7]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:61531 61531
   [junit4]   2> 220717 INFO  (Thread-286) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:61531 61531
   [junit4]   2> 221272 WARN  (Thread-286) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 70 /solr/aliases.json
   [junit4]   2> 70 /solr/clusterprops.json
   [junit4]   2> 13 /solr/configs/conf
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 70 /solr/clusterstate.json
   [junit4]   2> 4 /solr/collections/destination/state.json
   [junit4]   2> 4 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 4 /solr/collections/destinationCollection/state.json
   [junit4]   2> 4 /solr/collections/mainCorpus/state.json
   [junit4]   2> 2 /solr/collections/collection1_collection/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98160355200139276-127.0.0.1:61536_solr-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/98160355200139274-127.0.0.1:61534_solr-n_0000000001
   [junit4]   2> 2 /solr/collections/workQueue/state.json
   [junit4]   2> 2 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 70 /solr/live_nodes
   [junit4]   2> 70 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_EBB1030E59B136A7-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=FST50, join2_s=PostingsFormat(name=LuceneFixedGap), multiDefault=PostingsFormat(name=LuceneFixedGap), subject=PostingsFormat(name=LuceneFixedGap), price_f=PostingsFormat(name=LuceneFixedGap), a1_s=PostingsFormat(name=LuceneFixedGap), a_f=PostingsFormat(name=LuceneFixedGap), intDefault=PostingsFormat(name=Memory), join1_i=FST50, a2_s=BlockTreeOrds(blocksize=128), a_i=FST50, join3_i=BlockTreeOrds(blocksize=128), test_i=PostingsFormat(name=LuceneFixedGap), field_i=PostingsFormat(name=LuceneFixedGap), level1_s=FST50, body_t=BlockTreeOrds(blocksize=128), s_multi=BlockTreeOrds(blocksize=128), col_s=PostingsFormat(name=LuceneFixedGap), level2_s=PostingsFormat(name=Memory), a_ss=PostingsFormat(name=Memory), a_s=BlockTreeOrds(blocksize=128), a_t=PostingsFormat(name=Memory), id=FST50, text=BlockTreeOrds(blocksize=128), timestamp=PostingsFormat(name=Memory), i_multi=FST50, b_ls=PostingsFormat(name=LuceneFixedGap), test_dt=FST50, test_t=BlockTreeOrds(blocksize=128), ident_s=PostingsFormat(name=LuceneFixedGap), order_i=PostingsFormat(name=LuceneFixedGap), _version_=PostingsFormat(name=Memory), side_s=BlockTreeOrds(blocksize=128), miles_i=PostingsFormat(name=LuceneFixedGap)}, docValues:{expr_s=DocValuesFormat(name=Asserting), join2_s=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Lucene70), a_f=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Memory), join1_i=DocValuesFormat(name=Asserting), a2_s=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Asserting), join3_i=DocValuesFormat(name=Direct), test_i=DocValuesFormat(name=Lucene70), field_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Asserting), s_multi=DocValuesFormat(name=Direct), col_s=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Memory), a_ss=DocValuesFormat(name=Memory), a_s=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Memory), i_multi=DocValuesFormat(name=Asserting), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Asserting), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Memory), side_s=DocValuesFormat(name=Direct), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1847, maxMBSortInHeap=7.100250602344217, sim=RandomSimilarity(queryNorm=false): {}, locale=cs, timezone=MIT
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=202380424,total=305659904
   [junit4]   2> NOTE: All tests run in this JVM: [LargeVolumeJettyTest, GreaterThanEqualToEvaluatorTest, BasicHttpSolrClientTest, StreamExpressionParserTest, JdbcDriverTest, TestFastInputStream, GetByIdTest, ConcurrentUpdateSolrClientTest, SolrQueryTest, LBHttpSolrClientBuilderTest, CommonAdminParamsTest, FieldEvaluatorTest, HyperbolicCosineEvaluatorTest, SolrExampleBinaryTest, AndEvaluatorTest, ShardParamsTest, QueryResponseTest, TestToleratedUpdateError, ArcSineEvaluatorTest, OrEvaluatorTest, HyperbolicTangentEvaluatorTest, LessThanEqualToEvaluatorTest, LargeVolumeEmbeddedTest, TestSuggesterResponse, TestV2Request, AnlysisResponseBaseTest, SineEvaluatorTest, SolrExampleJettyTest, TestLBHttpSolrClient, SolrParamTest, FacetFieldTest, CloudSolrClientBuilderTest, CloudSolrClientCacheTest, CloudSolrClientTest, SelectWithEvaluatorsTest, StreamExpressionTest]
   [junit4] Completed [135/135 (1!)] on J1 in 76.27s, 89 tests, 1 error, 9 skipped <<< FAILURES!

[...truncated 42632 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4086 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4086/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

8 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([EDED2FDE535870B:5B8E3A6F49CC48FB]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1153)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:836)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
        at __randomizedtesting.SeedInfo.seed([EDED2FDE535870B:868AED274BC9EAF3]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:910)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ReplaceNodeTest

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([EDED2FDE535870B]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:171)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
        at org.apache.solr.cloud.MiniSolrCloudCluster.waitForAllNodes(MiniSolrCloudCluster.java:266)
        at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:260)
        at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
        at org.apache.solr.cloud.ReplaceNodeTest.setupCluster(ReplaceNodeTest.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1842)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:146)
        ... 31 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor65.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1839)
        ... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ReplaceNodeTest

Error Message:
ObjectTracker found 25 object(s) that were not released!!! [SolrZkClient, SolrZkClient, Overseer, InternalHttpClient, InternalHttpClient, ZkController, SolrZkClient, InternalHttpClient, SolrZkClient, ZkController, InternalHttpClient, SolrZkClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, ZkController, InternalHttpClient, InternalHttpClient, InternalHttpClient, ZkController, ZkController, InternalHttpClient, SolrZkClient, InternalHttpClient, ZkController] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.Overseer.start(Overseer.java:527)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:774)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)  at org.apache.solr.cloud.ZkController.init(ZkController.java:691)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:425)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 25 object(s) that were not released!!! [SolrZkClient, SolrZkClient, Overseer, InternalHttpClient, InternalHttpClient, ZkController, SolrZkClient, InternalHttpClient, SolrZkClient, ZkController, InternalHttpClient, SolrZkClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, ZkController, InternalHttpClient, InternalHttpClient, InternalHttpClient, ZkController, ZkController, InternalHttpClient, SolrZkClient, InternalHttpClient, ZkController]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.Overseer.start(Overseer.java:527)
        at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:774)
        at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
        at org.apache.solr.cloud.ZkController.init(ZkController.java:691)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:425)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)
        at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:94)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:289)
        at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:209)
        at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:501)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:427)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:185)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:301)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:177)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:382)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:245)
        at ja

[...truncated too long message...]

ch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 246541 INFO  (zkCallback-235-thread-1-processing-n:127.0.0.1:53452_solr) [n:127.0.0.1:53452_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 246542 INFO  (jetty-closer-216-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@79d82876{/solr,null,UNAVAILABLE}
   [junit4]   2> 246542 INFO  (coreCloseExecutor-738-thread-1) [n:127.0.0.1:53454_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=1267683926
   [junit4]   2> 246542 INFO  (zkCallback-234-thread-1-processing-n:127.0.0.1:53454_solr) [n:127.0.0.1:53454_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 246542 INFO  (zkCallback-235-thread-2-processing-n:127.0.0.1:53452_solr) [n:127.0.0.1:53452_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53452_solr
   [junit4]   2> 246543 WARN  (zkCallback-235-thread-1-processing-n:127.0.0.1:53452_solr) [n:127.0.0.1:53452_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 246544 INFO  (zkCallback-234-thread-1-processing-n:127.0.0.1:53454_solr) [n:127.0.0.1:53454_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 246544 INFO  (zkCallback-234-thread-2-processing-n:127.0.0.1:53454_solr) [n:127.0.0.1:53454_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53454_solr
   [junit4]   2> 246547 WARN  (zkCallback-234-thread-2-processing-n:127.0.0.1:53454_solr) [n:127.0.0.1:53454_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 246547 INFO  (jetty-closer-216-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@50f1631c{/solr,null,UNAVAILABLE}
   [junit4]   2> 248053 INFO  (jetty-closer-216-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44f951d0{/solr,null,UNAVAILABLE}
   [junit4]   2> 248054 ERROR (SUITE-StreamExpressionTest-seed#[A318309F9C2DCDEE]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 248055 INFO  (SUITE-StreamExpressionTest-seed#[A318309F9C2DCDEE]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53437 53437
   [junit4]   2> 248073 INFO  (Thread-281) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53437 53437
   [junit4]   2> 248210 WARN  (Thread-281) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 92 /solr/aliases.json
   [junit4]   2> 92 /solr/clusterprops.json
   [junit4]   2> 14 /solr/configs/conf
   [junit4]   2> 8 /solr/configs/ml
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 92 /solr/clusterstate.json
   [junit4]   2> 6 /solr/collections/destinationCollection/state.json
   [junit4]   2> 4 /solr/collections/destination/state.json
   [junit4]   2> 4 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 4 /solr/collections/mainCorpus/state.json
   [junit4]   2> 2 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/collections/workQueue/state.json
   [junit4]   2> 2 /solr/collections/uknownCollection/state.json
   [junit4]   2> 2 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 2 /solr/collections/modelCollection/state.json
   [junit4]   2> 2 /solr/collections/checkpointCollection/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98162271808651274-127.0.0.1:53452_solr-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 92 /solr/live_nodes
   [junit4]   2> 92 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_A318309F9C2DCDEE-001
   [junit4]   2> Jun 19, 2017 1:58:32 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=PostingsFormat(name=Asserting), name_s=PostingsFormat(name=Memory), iteration_i=PostingsFormat(name=Asserting), terms_ss=PostingsFormat(name=Memory), join2_s=PostingsFormat(name=Memory), multiDefault=PostingsFormat(name=Memory), subject=PostingsFormat(name=Memory), a_f=PostingsFormat(name=Memory), intDefault=Lucene50(blocksize=128), falseNegative_i=Lucene50(blocksize=128), a2_s=PostingsFormat(name=Memory), a_i=PostingsFormat(name=Asserting), level1_s=PostingsFormat(name=Asserting), body_t=PostingsFormat(name=Memory), a_s=PostingsFormat(name=Memory), a_t=Lucene50(blocksize=128), id=PostingsFormat(name=Asserting), text=PostingsFormat(name=Memory), alpha_d=PostingsFormat(name=Memory), i_multi=PostingsFormat(name=Asserting), idf_d=PostingsFormat(name=Memory), b_ls=PostingsFormat(name=Memory), checkpoint_ss=PostingsFormat(name=Memory), test_t=PostingsFormat(name=Memory), ident_s=PostingsFormat(name=Memory), order_i=PostingsFormat(name=Memory), error_d=PostingsFormat(name=Asserting), truePositive_i=PostingsFormat(name=Asserting), side_s=PostingsFormat(name=Memory), miles_i=PostingsFormat(name=Memory), field_s=Lucene50(blocksize=128), price_f=PostingsFormat(name=Memory), a1_s=PostingsFormat(name=Memory), join1_i=PostingsFormat(name=Asserting), join3_i=PostingsFormat(name=Memory), test_i=PostingsFormat(name=Memory), falsePositive_i=Lucene50(blocksize=128), field_i=PostingsFormat(name=Memory), s_multi=PostingsFormat(name=Memory), level2_s=Lucene50(blocksize=128), col_s=PostingsFormat(name=Memory), a_ss=Lucene50(blocksize=128), score_f=PostingsFormat(name=Memory), tv_text=Lucene50(blocksize=128), term_s=PostingsFormat(name=Memory), text_s=PostingsFormat(name=Asserting), timestamp=Lucene50(blocksize=128), idfs_ds=Lucene50(blocksize=128), weights_ds=PostingsFormat(name=Memory), featureSet_s=PostingsFormat(name=Memory), trueNegative_i=PostingsFormat(name=Memory), index_i=PostingsFormat(name=Memory), test_dt=PostingsFormat(name=Asserting), out_i=PostingsFormat(name=Memory), whitetok=PostingsFormat(name=Memory), _version_=Lucene50(blocksize=128)}, docValues:{expr_s=DocValuesFormat(name=Lucene70), name_s=DocValuesFormat(name=Direct), iteration_i=DocValuesFormat(name=Lucene70), terms_ss=DocValuesFormat(name=Lucene70), field_s=DocValuesFormat(name=Asserting), join2_s=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Lucene70), a_f=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Asserting), falseNegative_i=DocValuesFormat(name=Asserting), join1_i=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Lucene70), join3_i=DocValuesFormat(name=Direct), test_i=DocValuesFormat(name=Lucene70), falsePositive_i=DocValuesFormat(name=Asserting), field_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Lucene70), s_multi=DocValuesFormat(name=Direct), level2_s=DocValuesFormat(name=Asserting), col_s=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Asserting), a_s=DocValuesFormat(name=Direct), score_f=DocValuesFormat(name=Direct), term_s=DocValuesFormat(name=Lucene70), text_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Asserting), i_multi=DocValuesFormat(name=Lucene70), featureSet_s=DocValuesFormat(name=Lucene70), trueNegative_i=DocValuesFormat(name=Direct), index_i=DocValuesFormat(name=Lucene70), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Lucene70), out_i=DocValuesFormat(name=Lucene70), checkpoint_ss=DocValuesFormat(name=Direct), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Asserting), truePositive_i=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Direct), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=790, maxMBSortInHeap=5.162787186906376, sim=RandomSimilarity(queryNorm=false): {}, locale=is-IS, timezone=Australia/Yancowinna
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=108305512,total=294801408
   [junit4]   2> NOTE: All tests run in this JVM: [HyperbolicTangentEvaluatorTest, QueryResponseTest, LBHttpSolrClientBuilderTest, JdbcDriverTest, AnlysisResponseBaseTest, SolrExampleBinaryTest, StreamExpressionToExplanationTest, SolrQueryTest, ClientUtilsTest, ShardParamsTest, CoalesceEvaluatorTest, SineEvaluatorTest, TestRetryUtil, CloudSolrClientCacheTest, CubedRootEvaluatorTest, AndEvaluatorTest, SolrParamTest, SolrPingTest, DocumentAnalysisResponseTest, SolrExampleStreamingBinaryTest, ArcSineEvaluatorTest, TestLBHttpSolrClient, EqualsEvaluatorTest, ContentStreamTest, TestFastInputStream, LargeVolumeBinaryJettyTest, SubtractEvaluatorTest, TestJsonRecordReader, BasicHttpSolrClientTest, TestNamedListCodec, LessThanEqualToEvaluatorTest, TestSolrProperties, MergeIndexesEmbeddedTest, FacetFieldTest, GetByIdTest, CloudSolrClientBuilderTest, CloudSolrClientTest, JDBCStreamTest, SelectWithEvaluatorsTest, StreamExpressionTest]
   [junit4] Completed [135/135 (1!)] on J1 in 113.69s, 89 tests, 1 error <<< FAILURES!

[...truncated 42631 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4087 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4087/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testArraySort

Error Message:
--> http://127.0.0.1:64938/solr/collection1:Invalid stream expression arraySort(array(11.5,12.3,4,3,1,0)) - function 'arraySort' is unknown (not mapped to a valid TupleStream)

Stack Trace:
java.io.IOException: --> http://127.0.0.1:64938/solr/collection1:Invalid stream expression arraySort(array(11.5,12.3,4,3,1,0)) - function 'arraySort' is unknown (not mapped to a valid TupleStream)
        at __randomizedtesting.SeedInfo.seed([C2A06E95E4E12915:F416B8B86338D56C]:0)
        at org.apache.solr.client.solrj.io.stream.SolrStream.read(SolrStream.java:219)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.getTuples(StreamExpressionTest.java:7487)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testArraySort(StreamExpressionTest.java:5840)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([527BDC2F19F3C004:3016226ED67DA03A]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.junit.Assert.assertNotNull(Assert.java:537)
        at org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter(MetricsHandlerTest.java:201)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12795 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.MetricsHandlerTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.admin.MetricsHandlerTest_527BDC2F19F3C004-001/init-core-data-001
   [junit4]   2> 2324203 WARN  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 2324204 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 2324207 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2324207 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 2324208 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib, /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 2324243 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2324257 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 2324323 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2324386 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a034ff4
   [junit4]   2> 2324388 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a034ff4
   [junit4]   2> 2324388 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a034ff4
   [junit4]   2> 2324390 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib, /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 2324431 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2324445 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2324516 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2324521 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 2324521 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a034ff4
   [junit4]   2> 2324521 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2324521 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.admin.MetricsHandlerTest_527BDC2F19F3C004-001/init-core-data-001/]
   [junit4]   2> 2324522 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=26.4990234375, floorSegmentMB=0.306640625, forceMergeDeletesPctAllowed=26.112989000434577, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2324601 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2324601 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2324602 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2324602 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2324602 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=14, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.889428508619749]
   [junit4]   2> 2324602 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@32b69ed5[collection1] main]
   [junit4]   2> 2324603 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2324613 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2324628 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2324628 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2324628 INFO  (coreLoadExecutor-10860-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570626071452188672
   [junit4]   2> 2324629 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 2324629 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 2324629 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 2324630 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 2324633 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 2324635 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 2324639 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 2324640 INFO  (TEST-MetricsHandlerTest.testPropertyFilter-seed#[527BDC2F19F3C004]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPropertyFilter
   [junit4]   2> 2324645 INFO  (TEST-MetricsHandlerTest.testPropertyFilter-seed#[527BDC2F19F3C004]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPropertyFilter
   [junit4]   2> 2324646 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MetricsHandlerTest -Dtests.method=testPropertyFilter -Dtests.seed=527BDC2F19F3C004 -Dtests.slow=true -Dtests.locale=bg-BG -Dtests.timezone=Asia/Khandyga -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.02s J0 | MetricsHandlerTest.testPropertyFilter <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([527BDC2F19F3C004:3016226ED67DA03A]:0)
   [junit4]    > at org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter(MetricsHandlerTest.java:201)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2324649 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 2324651 INFO  (searcherExecutor-10861-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@32b69ed5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2324652 INFO  (TEST-MetricsHandlerTest.test-seed#[527BDC2F19F3C004]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2324673 INFO  (TEST-MetricsHandlerTest.test-seed#[527BDC2F19F3C004]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2324730 INFO  (TEST-MetricsHandlerTest.test-seed#[527BDC2F19F3C004]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> 2324734 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[527BDC2F19F3C004]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCompact
   [junit4]   2> 2324746 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[527BDC2F19F3C004]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCompact
   [junit4]   2> 2324746 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 2324746 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=655985693
   [junit4]   2> 2324746 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2324747 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2324748 INFO  (SUITE-MetricsHandlerTest-seed#[527BDC2F19F3C004]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2324749 INFO  (coreCloseExecutor-10866-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7781d594
   [junit4]   2> 2324749 INFO  (coreCloseExecutor-10866-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2004997524
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.admin.MetricsHandlerTest_527BDC2F19F3C004-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=bg-BG, timezone=Asia/Khandyga
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=136673728,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [SuggestComponentContextFilterQueryTest, BasicAuthStandaloneTest, ClassificationUpdateProcessorFactoryTest, CollectionReloadTest, TestPhraseSuggestions, TestDocBasedVersionConstraints, TestPushWriter, CursorPagingTest, TestBinaryField, SSLMigrationTest, TestLocalFSCloudBackupRestore, DistribCursorPagingTest, TestBackupRepositoryFactory, TestSolrDeletionPolicy1, TestExtendedDismaxParser, TestDistributedStatsComponentCardinality, ImplicitSnitchTest, FileUtilsTest, TestCollapseQParserPlugin, TestRandomDVFaceting, DistributedFacetPivotSmallTest, TestBulkSchemaConcurrent, CdcrVersionReplicationTest, TestTestInjection, TestWordDelimiterFilterFactory, TestPayloadScoreQParserPlugin, IndexBasedSpellCheckerTest, DirectUpdateHandlerTest, TestNamedUpdateProcessors, TestSolrCloudWithDelegationTokens, ZkControllerTest, TestManagedStopFilterFactory, TestChildDocTransformer, TestNumericTerms64, PingRequestHandlerTest, DistributedVersionInfoTest, TestUpdate, TestTlogReplica, TestPerFieldSimilarity, DeleteInactiveReplicaTest, PeerSyncTest, TestXIncludeConfig, TestHashPartitioner, TestNumericRangeQuery64, TestSchemaNameResource, InfixSuggestersTest, TestSystemIdResolver, TestSolrCloudWithSecureImpersonation, SolrXmlInZkTest, JSONWriterTest, TestFieldTypeCollectionResource, MBeansHandlerTest, TestLegacyFieldCache, DirectoryFactoryTest, TestBM25SimilarityFactory, BinaryUpdateRequestHandlerTest, TestDistributedSearch, TestReRankQParserPlugin, TestSimpleTextCodec, TestSlowCompositeReaderWrapper, TestMiniSolrCloudCluster, InfoHandlerTest, TestMultiWordSynonyms, TestManagedResource, TestInPlaceUpdatesStandalone, TestReqParamsAPI, UniqFieldsUpdateProcessorFactoryTest, TestManagedSynonymGraphFilterFactory, TestHighlightDedupGrouping, TestReloadDeadlock, BasicDistributedZkTest, TestCloudPseudoReturnFields, TestRebalanceLeaders, TestConfigSetsAPI, TestHdfsBackupRestoreCore, OverseerRolesTest, TestMultiValuedNumericRangeQuery, TestUseDocValuesAsStored, TestTolerantSearch, TestQuerySenderNoQuery, TestCoreContainer, TestXmlQParser, RecoveryZkTest, PrimitiveFieldTypeTest, TestSortByMinMaxFunction, TestStressUserVersions, CdcrUpdateLogTest, TestManagedSchema, TestSolrCoreSnapshots, HighlighterMaxOffsetTest, BlockJoinFacetSimpleTest, TestSolrJ, HLLSerializationTest, TestSystemCollAutoCreate, ZkStateReaderTest, ShardRoutingTest, OverseerTest, OpenExchangeRatesOrgProviderTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestLazyCores, SolrPluginUtilsTest, TestPartialUpdateDeduplication, LeaderFailureAfterFreshStartTest, OutputWriterTest, DocExpirationUpdateProcessorFactoryTest, TestSchemaManager, TestSQLHandler, StatelessScriptUpdateProcessorFactoryTest, StandardRequestHandlerTest, BlockJoinFacetDistribTest, RAMDirectoryFactoryTest, TestUtils, AutoScalingHandlerTest, TestFieldSortValues, TestJsonRequest, HighlighterTest, TestInitQParser, PathHierarchyTokenizerFactoryTest, TestBinaryResponseWriter, TestRawTransformer, TestAuthorizationFramework, LeaderInitiatedRecoveryOnShardRestartTest, SearchHandlerTest, TestTrackingShardHandlerFactory, QueryEqualityTest, MoreLikeThisHandlerTest, SuggesterFSTTest, TestDefaultStatsCache, TestImplicitCoreProperties, TestInitParams, HdfsBasicDistributedZk2Test, SchemaVersionSpecificBehaviorTest, TestConfigsApi, TestCSVLoader, AddBlockUpdateTest, QueryElevationComponentTest, TestCharFilters, TriLevelCompositeIdRoutingTest, SortSpecParsingTest, DisMaxRequestHandlerTest, V2StandaloneTest, TestDownShardTolerantSearch, TestSerializedLuceneMatchVersion, SimpleMLTQParserTest, TestClassNameShortening, TestPseudoReturnFields, TestSchemaSimilarityResource, RulesTest, DateMathParserTest, CursorMarkTest, HardAutoCommitTest, FastVectorHighlighterTest, DistributedFacetExistsSmallTest, HdfsLockFactoryTest, ReplaceNodeTest, CdcrBootstrapTest, MissingSegmentRecoveryTest, PolyFieldTest, TestMinMaxOnMultiValuedField, CollectionsAPISolrJTest, TestDFISimilarityFactory, TestBlobHandler, DeleteShardTest, SegmentsInfoRequestHandlerTest, TestRTGBase, SpellCheckCollatorWithCollapseTest, TestConfigSets, HdfsUnloadDistributedZkTest, SoftAutoCommitTest, TestSimpleQParserPlugin, ConnectionReuseTest, UnloadDistributedZkTest, ZkNodePropsTest, SolrCLIZkUtilsTest, TestLRUStatsCache, TestRandomFlRTGCloud, BasicZkTest, TestReplicationHandler, TestZkChroot, ZkCLITest, TestFaceting, DistributedSpellCheckComponentTest, TestReload, DistributedTermsComponentTest, StatsComponentTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SolrInfoBeanTest, TestSurroundQueryParser, LukeRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TermsComponentTest, RequestHandlersTest, TestValueSourceCache, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, TestCollationField, TestCSVResponseWriter, TestLFUCache, DirectSolrConnectionTest, SynonymTokenizerTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestFastWriter, ResourceLoaderTest, ScriptEngineTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionTooManyReplicasTest, ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteReplicaTest, DistributedQueueTest, LeaderElectionContextKeyTest, MultiThreadedOCPTest, NodeMutatorTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerTaskQueueTest, PeerSyncReplicationTest, RecoveryAfterSoftCommitTest, RestartWhileUpdatingTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCloudPivotFacet, TestCloudRecovery, TestClusterProperties, TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, TestHdfsCloudBackupRestore, TestLeaderElectionZkExpiry, TestMiniSolrCloudClusterSSL, TestRandomRequestDistribution, TestRequestForwarding, TestRequestStatusCollectionAPI, TestSizeLimitedDistributedMap, TestTolerantUpdateProcessorRandomCloud, VMParamsZkACLAndCredentialsProvidersTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest, HdfsNNFailoverTest, HdfsRecoverLeaseTest, HdfsRestartWhileUpdatingTest, HdfsSyncSliceTest, HdfsThreadLeakTest, HdfsTlogReplayBufferedWhileIndexingTest, StressHdfsTest, ZkStateWriterTest, BlobRepositoryCloudTest, CachingDirectoryFactoryTest, ConfigureRecoveryStrategyTest, ExitableDirectoryReaderTest, TestConfigSetImmutable, TestDynamicLoading, RequestLoggingTest, TestRestoreCore, TestSolrConfigHandlerConcurrent, CoreAdminCreateDiscoverTest, CoreAdminRequestStatusTest, MetricsHandlerTest]
   [junit4] Completed [591/725 (1!)] on J0 in 0.58s, 3 tests, 1 failure <<< FAILURES!

[...truncated 1219 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.StreamExpressionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/init-core-data-001
   [junit4]   2> 171925 WARN  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=19 numCloses=19
   [junit4]   2> 171925 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 171931 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 171933 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001
   [junit4]   2> 171933 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 171933 INFO  (Thread-685) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 171933 INFO  (Thread-685) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 171937 ERROR (Thread-685) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 172037 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64936
   [junit4]   2> 172063 INFO  (jetty-launcher-624-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 172063 INFO  (jetty-launcher-624-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 172063 INFO  (jetty-launcher-624-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 172064 INFO  (jetty-launcher-624-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 172066 INFO  (jetty-launcher-624-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5c5942b2{/solr,null,AVAILABLE}
   [junit4]   2> 172068 INFO  (jetty-launcher-624-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6e287622{HTTP/1.1,[http/1.1]}{127.0.0.1:64938}
   [junit4]   2> 172068 INFO  (jetty-launcher-624-thread-1) [    ] o.e.j.s.Server Started @177716ms
   [junit4]   2> 172068 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64938}
   [junit4]   2> 172069 ERROR (jetty-launcher-624-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 172069 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 172069 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 172069 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 172069 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T09:56:45.484Z
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1fa45322{/solr,null,AVAILABLE}
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41b8e572{/solr,null,AVAILABLE}
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@55a0fe14{HTTP/1.1,[http/1.1]}{127.0.0.1:64939}
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.e.j.s.Server Started @177718ms
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64939}
   [junit4]   2> 172070 ERROR (jetty-launcher-624-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 172070 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T09:56:45.485Z
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62465fbf{/solr,null,AVAILABLE}
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f71e18d{HTTP/1.1,[http/1.1]}{127.0.0.1:64942}
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-3) [    ] o.e.j.s.Server Started @177721ms
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64942}
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f79f490{HTTP/1.1,[http/1.1]}{127.0.0.1:64940}
   [junit4]   2> 172072 ERROR (jetty-launcher-624-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-4) [    ] o.e.j.s.Server Started @177721ms
   [junit4]   2> 172072 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64940}
   [junit4]   2> 172073 ERROR (jetty-launcher-624-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T09:56:45.488Z
   [junit4]   2> 172073 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T09:56:45.488Z
   [junit4]   2> 172074 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 172076 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 172078 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 172079 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 172088 INFO  (jetty-launcher-624-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64936/solr
   [junit4]   2> 172088 INFO  (jetty-launcher-624-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64936/solr
   [junit4]   2> 172091 INFO  (jetty-launcher-624-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64936/solr
   [junit4]   2> 172093 INFO  (jetty-launcher-624-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64936/solr
   [junit4]   2> 172137 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 172137 INFO  (jetty-launcher-624-thread-1) [n:127.0.0.1:64938_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 172138 INFO  (jetty-launcher-624-thread-4) [n:127.0.0.1:64940_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 172139 INFO  (jetty-launcher-624-thread-2) [n:127.0.0.1:64939_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 172139 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64942_solr
   [junit4]   2> 172141 INFO  (jetty-launcher-624-thread-4) [n:127.0.0.1:64940_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64940_solr
   [junit4]   2> 172142 INFO  (jetty-launcher-624-thread-1) [n:127.0.0.1:64938_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64938_solr
   [junit4]   2> 172142 INFO  (jetty-launcher-624-thread-2) [n:127.0.0.1:64939_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64939_solr
   [junit4]   2> 172142 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.c.Overseer Overseer (id=98164159633883147-127.0.0.1:64942_solr-n_0000000000) starting
   [junit4]   2> 172144 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 172145 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 172145 INFO  (zkCallback-645-thread-1-processing-n:127.0.0.1:64942_solr) [n:127.0.0.1:64942_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 172146 INFO  (zkCallback-643-thread-1-processing-n:127.0.0.1:64939_solr) [n:127.0.0.1:64939_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 172148 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 172148 INFO  (zkCallback-645-thread-1-processing-n:127.0.0.1:64942_solr) [n:127.0.0.1:64942_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 172148 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 172148 INFO  (zkCallback-643-thread-1-processing-n:127.0.0.1:64939_solr) [n:127.0.0.1:64939_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 172176 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64942_solr
   [junit4]   2> 172178 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 172178 INFO  (zkCallback-645-thread-1-processing-n:127.0.0.1:64942_solr) [n:127.0.0.1:64942_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 172178 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 172178 INFO  (zkCallback-643-thread-1-processing-n:127.0.0.1:64939_solr) [n:127.0.0.1:64939_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 172353 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64942.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172367 INFO  (jetty-launcher-624-thread-1) [n:127.0.0.1:64938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64938.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172371 INFO  (jetty-launcher-624-thread-1) [n:127.0.0.1:64938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64938.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172371 INFO  (jetty-launcher-624-thread-1) [n:127.0.0.1:64938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64938.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172372 INFO  (jetty-launcher-624-thread-1) [n:127.0.0.1:64938_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node1/.
   [junit4]   2> 172392 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64942.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172392 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64942.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172393 INFO  (jetty-launcher-624-thread-3) [n:127.0.0.1:64942_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node3/.
   [junit4]   2> 172459 INFO  (jetty-launcher-624-thread-4) [n:127.0.0.1:64940_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64940.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172461 INFO  (jetty-launcher-624-thread-4) [n:127.0.0.1:64940_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64940.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172461 INFO  (jetty-launcher-624-thread-4) [n:127.0.0.1:64940_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64940.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172462 INFO  (jetty-launcher-624-thread-4) [n:127.0.0.1:64940_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node4/.
   [junit4]   2> 172595 INFO  (jetty-launcher-624-thread-2) [n:127.0.0.1:64939_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64939.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172600 INFO  (jetty-launcher-624-thread-2) [n:127.0.0.1:64939_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64939.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172600 INFO  (jetty-launcher-624-thread-2) [n:127.0.0.1:64939_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64939.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 172601 INFO  (jetty-launcher-624-thread-2) [n:127.0.0.1:64939_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node2/.
   [junit4]   2> 172662 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 172671 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64936/solr ready
   [junit4]   2> 172690 INFO  (qtp837992558-2981) [n:127.0.0.1:64942_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 172699 INFO  (OverseerThreadFactory-1022-thread-1-processing-n:127.0.0.1:64942_solr) [n:127.0.0.1:64942_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 172818 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 172819 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 172819 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 172820 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 172940 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 172940 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 173860 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 173860 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 173872 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
   [junit4]   2> 173888 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 173989 WARN  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 173989 WARN  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 173990 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 173990 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 173991 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64938.solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 173991 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 173991 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node1/collection1_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node1/./collection1_shard1_replica_n1/data/]
   [junit4]   2> 174006 WARN  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 174006 WARN  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 174007 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 174007 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 174007 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64940.solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@32e95659
   [junit4]   2> 174007 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 174007 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node4/collection1_shard2_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001/tempDir-001/node4/./collection1_shard2_replica_n1/data/]
   [junit4]   2> 174153 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 174153 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 174155 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 174155 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 174157 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 174157 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 174159 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 174159 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 174182 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@31054b01[collection1_shard1_replica_n1] main]
   [junit4]   2> 174184 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 174185 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 174186 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 174186 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570626556501426176
   [junit4]   2> 174187 INFO  (searcherExecutor-1031-thread-1-processing-n:127.0.0.1:64938_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@31054b01[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 174195 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@e24061c[collection1_shard2_replica_n1] main]
   [junit4]   2> 174197 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 174198 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 174199 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 174199 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 174199 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/
   [junit4]   2> 174200 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 174200 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 174200 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 174200 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 174202 INFO  (searcherExecutor-1032-thread-1-processing-n:127.0.0.1:64940_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@e24061c[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 174202 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570626556518203392
   [junit4]   2> 174208 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/ shard1
   [junit4]   2> 174212 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 174212 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 174212 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/
   [junit4]   2> 174212 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 174212 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/ has no replicas
   [junit4]   2> 174212 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 174218 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 174330 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 174330 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 174333 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 174336 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1518
   [junit4]   2> 174373 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 174378 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1559
   [junit4]   2> 174384 INFO  (qtp837992558-2981) [n:127.0.0.1:64942_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 174481 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 174481 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 174703 INFO  (OverseerCollectionConfigSetProcessor-98164159633883147-127.0.0.1:64942_solr-n_0000000000) [n:127.0.0.1:64942_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 175393 INFO  (qtp837992558-2981) [n:127.0.0.1:64942_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2702
   [junit4]   2> 175393 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):90
   [junit4]   2> 175394 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
   [junit4]   2> 175422 INFO  (TEST-StreamExpressionTest.testCovariance-seed#[C2A06E95E4E12915]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCovariance
   [junit4]   2> 175438 INFO  (qtp1691193282-2993) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1570626557802708992)} 0 11
   [junit4]   2> 175440 INFO  (qtp1691193282-2996) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626557816340480,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 175440 INFO  (qtp1691193282-2996) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 175441 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626557817389056,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 175441 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 175443 INFO  (qtp1691193282-2996) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 175443 INFO  (qtp1691193282-2996) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
   [junit4]   2> 175444 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 175444 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
   [junit4]   2> 175444 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1570626557800611840),commit=} 0 19
   [junit4]   2> 175474 INFO  (qtp1892919422-2974) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&wt=javabin&version=2}{add=[id_2 (1570626557822631936), id_10 (1570626557838360576), id_13 (1570626557838360577), id_14 (1570626557838360578), id_16 (1570626557839409152), id_17 (1570626557839409153), id_18 (1570626557839409154), id_21 (1570626557839409155), id_22 (1570626557839409156), id_23 (1570626557839409157), ... (125 adds)]} 0 28
   [junit4]   2> 175485 INFO  (qtp1691193282-2997) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&wt=javabin&version=2}{add=[id_1 (1570626557823680512), id_3 (1570626557854089216), id_4 (1570626557854089217), id_5 (1570626557854089218), id_6 (1570626557854089219), id_7 (1570626557854089220), id_8 (1570626557854089221), id_9 (1570626557854089222), id_11 (1570626557854089223), id_12 (1570626557854089224), ... (125 adds)]} 0 38
   [junit4]   2> 175487 INFO  (qtp1892919422-2965) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626557865623552,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 175487 INFO  (qtp1892919422-2965) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4679426f commitCommandVersion:1570626557865623552
   [junit4]   2> 175487 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626557865623552,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 175487 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4348fc69 commitCommandVersion:1570626557865623552
   [junit4]   2> 175952 INFO  (qtp1892919422-2965) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1822d1f8[collection1_shard1_replica_n1] main]
   [junit4]   2> 175954 INFO  (searcherExecutor-1031-thread-1-processing-n:127.0.0.1:64938_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@1822d1f8[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C125)))}
   [junit4]   2> 175962 INFO  (qtp1892919422-2965) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 175962 INFO  (qtp1892919422-2965) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 475
   [junit4]   2> 176009 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@11eab74a[collection1_shard2_replica_n1] main]
   [junit4]   2> 176009 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 176010 INFO  (searcherExecutor-1032-thread-1-processing-n:127.0.0.1:64940_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@11eab74a[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C125)))}
   [junit4]   2> 176010 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 523
   [junit4]   2> 176010 INFO  (qtp1892919422-2975) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&commit=true&wt=javabin&version=2}{commit=} 0 524
   [junit4]   2> 176016 INFO  (qtp1892919422-2970) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/stream params={expr=let(a%3Dtimeseries(collection1,+q%3D"*:*",+start%3D"2013-01-01T01:00:00.000Z",+end%3D"2016-12-01T01:00:00.000Z",+gap%3D"%2B1YEAR",+field%3D"test_dt",+count(*),+sum(price_f),+max(price_f),+min(price_f)),+b%3Dselect(timeseries(collection1,+q%3D"*:*",+start%3D"2013-01-01T01:00:00.000Z",+end%3D"2016-12-01T01:00:00.000Z",+gap%3D"%2B1YEAR",+field%3D"test_dt",+count(*),+sum(price_f),+max(price_f),+min(price_f)),mult(-1,+count(*))+as+nvalue),+c%3Dcol(a,+count(*)),+d%3Dcol(b,+nvalue),+tuple(colc%3Dc,+cold%3Dd,+cov%3Dcov(c,d)))&wt=json&version=2.2} status=0 QTime=3
   [junit4]   2> 176021 INFO  (qtp1892919422-2970) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 176022 INFO  (qtp1892919422-2970) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64936/solr ready
   [junit4]   2> 176125 INFO  (qtp1691193282-2993) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:3&_facet_={}&fl=id&fl=score&shards.purpose=1048580&start=0&fsv=true&shard.url=http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&field=test_dt&NOW=1497866209439&isShard=true&wt=javabin} hits=125 status=0 QTime=71
   [junit4]   2> 176126 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:3&_facet_={}&fl=id&fl=score&shards.purpose=1048580&start=0&fsv=true&shard.url=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&field=test_dt&NOW=1497866209439&isShard=true&wt=javabin} hits=125 status=0 QTime=72
   [junit4]   2> 176127 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&_stateVer_=collection1:3&field=test_dt&rows=0&wt=javabin&version=2} hits=250 status=0 QTime=103
   [junit4]   2> 176134 INFO  (qtp1892919422-2965) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:3&_facet_={}&fl=id&fl=score&shards.purpose=1048580&start=0&fsv=true&shard.url=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&field=test_dt&NOW=1497866209544&isShard=true&wt=javabin} hits=125 status=0 QTime=2
   [junit4]   2> 176136 INFO  (qtp1691193282-2997) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:3&_facet_={}&fl=id&fl=score&shards.purpose=1048580&start=0&fsv=true&shard.url=http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&field=test_dt&NOW=1497866209544&isShard=true&wt=javabin} hits=125 status=0 QTime=3
   [junit4]   2> 176137 INFO  (qtp1892919422-2974) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&_stateVer_=collection1:3&field=test_dt&rows=0&wt=javabin&version=2} hits=250 status=0 QTime=7
   [junit4]   2> 176235 INFO  (TEST-StreamExpressionTest.testCovariance-seed#[C2A06E95E4E12915]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCovariance
   [junit4]   2> 176272 INFO  (TEST-StreamExpressionTest.testTupleStream-seed#[C2A06E95E4E12915]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTupleStream
   [junit4]   2> 176276 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6eab35e4[collection1_shard2_replica_n1] realtime]
   [junit4]   2> 176276 INFO  (qtp1892919422-2975) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@314957a5[collection1_shard1_replica_n1] realtime]
   [junit4]   2> 176277 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1570626558690852864)} 0 3
   [junit4]   2> 176278 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626558695047168,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 176278 INFO  (qtp1691193282-2969) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626558695047168,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 176278 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4679426f commitCommandVersion:1570626558695047168
   [junit4]   2> 176278 INFO  (qtp1691193282-2969) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4348fc69 commitCommandVersion:1570626558695047168
   [junit4]   2> 176300 INFO  (qtp1691193282-2969) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@352ccf64[collection1_shard2_replica_n1] main]
   [junit4]   2> 176300 INFO  (qtp1691193282-2969) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 176301 INFO  (searcherExecutor-1032-thread-1-processing-n:127.0.0.1:64940_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@352ccf64[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 176302 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3d93be72[collection1_shard1_replica_n1] main]
   [junit4]   2> 176303 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 176304 INFO  (searcherExecutor-1031-thread-1-processing-n:127.0.0.1:64938_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@3d93be72[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 176305 INFO  (qtp1892919422-2973) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 27
   [junit4]   2> 176305 INFO  (qtp1691193282-2969) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64938/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 27
   [junit4]   2> 176306 INFO  (qtp1892919422-2975) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1570626558689804288),commit=} 0 33
   [junit4]   2> 176310 INFO  (qtp1691193282-2995) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&wt=javabin&version=2}{add=[hello (1570626558724407296), hello1 (1570626558724407297)]} 0 4
   [junit4]   2> 176313 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626558731747328,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 176313 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1570626558731747328,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 176313 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4348fc69 commitCommandVersion:1570626558731747328
   [junit4]   2> 176313 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 176313 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 176313 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
   [junit4]   2> 176347 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@678daf4a[collection1_shard2_replica_n1] main]
   [junit4]   2> 176347 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 176348 INFO  (searcherExecutor-1032-thread-1-processing-n:127.0.0.1:64940_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@678daf4a[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.0.0):C2)))}
   [junit4]   2> 176348 INFO  (qtp1691193282-2992) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 35
   [junit4]   2> 176348 INFO  (qtp1691193282-2993) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:3&commit=true&wt=javabin&version=2}{commit=} 0 37
   [junit4]   2> 176353 INFO  (qtp1892919422-2974) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/stream params={expr=tuple(results%3Dsearch(collection1,+q%3D"*:*",+fl%3D"id,test_t",+sort%3D"id+desc"),+sum%3Dadd(1,1))&wt=json&version=2.2} status=0 QTime=0
   [junit4]   2> 176358 INFO  (qtp1691193282-2996) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,test_t&sort=id+desc&wt=json&version=2.2} hits=2 status=0 QTime=0
   [junit4]   2> 176363 INFO  (qtp1892919422-2968) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,test_t&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 176365 INFO  (TEST-StreamExpressionTest.testTupleStream-seed#[C2A06E95E4E12915]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testTupleStream
   [junit4]   2> 176403 INFO  (TEST-StreamExpressionTest.testTopicStream-seed#[C2A06E95E4E12915]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTopicStream
   [junit4]   2> 176410 INFO  (qtp1691193282-2998) [n:127.0.0.1:64940_solr c:collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@43132eff[collection1_shard2_replica_n1] realtime]
   [junit4]   2> 176417 INFO  (qtp1892919422-2972) [n:127.0.0.1:64938_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:64940/solr/collection1_shard2_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1570626558828216320)} 0 12
   [junit4]   2> 176419 INFO  (qtp1691193282-2995) [n:127.0.0.1:64940_solr c:collection1 s:sha

[...truncated too long message...]

solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 281139 INFO  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (2)
   [junit4]   2> 281139 INFO  (jetty-closer-625-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@62465fbf{/solr,null,UNAVAILABLE}
   [junit4]   2> 281140 INFO  (zkCallback-644-thread-2-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64938_solr
   [junit4]   2> 281144 WARN  (zkCallback-646-thread-1-processing-n:127.0.0.1:64940_solr) [n:127.0.0.1:64940_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 281144 INFO  (jetty-closer-625-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@41b8e572{/solr,null,UNAVAILABLE}
   [junit4]   2> 281145 INFO  (zkCallback-644-thread-2-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 281145 WARN  (zkCallback-644-thread-2-processing-n:127.0.0.1:64938_solr) [n:127.0.0.1:64938_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 281146 INFO  (jetty-closer-625-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5c5942b2{/solr,null,UNAVAILABLE}
   [junit4]   2> 281148 ERROR (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 281148 INFO  (SUITE-StreamExpressionTest-seed#[C2A06E95E4E12915]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64936 64936
   [junit4]   2> 281158 INFO  (Thread-685) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64936 64936
   [junit4]   2> 281246 WARN  (Thread-685) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 89 /solr/aliases.json
   [junit4]   2> 89 /solr/clusterprops.json
   [junit4]   2> 12 /solr/configs/conf
   [junit4]   2> 8 /solr/configs/ml
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 89 /solr/clusterstate.json
   [junit4]   2> 6 /solr/collections/destinationCollection/state.json
   [junit4]   2> 4 /solr/collections/destination/state.json
   [junit4]   2> 4 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 4 /solr/collections/mainCorpus/state.json
   [junit4]   2> 2 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/collections/workQueue/state.json
   [junit4]   2> 2 /solr/collections/uknownCollection/state.json
   [junit4]   2> 2 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 2 /solr/collections/modelCollection/state.json
   [junit4]   2> 2 /solr/collections/checkpointCollection/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 89 /solr/live_nodes
   [junit4]   2> 89 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_C2A06E95E4E12915-001
   [junit4]   2> Jun 19, 2017 9:58:34 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=PostingsFormat(name=Memory), name_s=PostingsFormat(name=MockRandom), iteration_i=PostingsFormat(name=MockRandom), terms_ss=PostingsFormat(name=Direct), join2_s=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=Direct), subject=PostingsFormat(name=Direct), intDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_f=PostingsFormat(name=LuceneVarGapDocFreqInterval), a2_s=PostingsFormat(name=Memory), falseNegative_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_i=PostingsFormat(name=MockRandom), level1_s=PostingsFormat(name=Memory), body_t=PostingsFormat(name=MockRandom), a_s=PostingsFormat(name=Memory), a_t=PostingsFormat(name=Direct), id=PostingsFormat(name=MockRandom), text=PostingsFormat(name=Memory), alpha_d=PostingsFormat(name=MockRandom), i_multi=PostingsFormat(name=MockRandom), idf_d=PostingsFormat(name=LuceneVarGapDocFreqInterval), b_ls=PostingsFormat(name=LuceneVarGapDocFreqInterval), checkpoint_ss=PostingsFormat(name=MockRandom), test_t=PostingsFormat(name=MockRandom), ident_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), order_i=PostingsFormat(name=Direct), error_d=PostingsFormat(name=Memory), side_s=PostingsFormat(name=MockRandom), truePositive_i=PostingsFormat(name=Memory), miles_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), field_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), price_f=PostingsFormat(name=Direct), a1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), join1_i=PostingsFormat(name=Memory), join3_i=PostingsFormat(name=MockRandom), test_i=PostingsFormat(name=Direct), falsePositive_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), field_i=PostingsFormat(name=Direct), s_multi=PostingsFormat(name=Memory), col_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_ss=PostingsFormat(name=Direct), level2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), score_f=PostingsFormat(name=Memory), tv_text=PostingsFormat(name=LuceneVarGapDocFreqInterval), term_s=PostingsFormat(name=Direct), text_s=PostingsFormat(name=Memory), timestamp=PostingsFormat(name=Direct), idfs_ds=PostingsFormat(name=Direct), weights_ds=PostingsFormat(name=MockRandom), featureSet_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), trueNegative_i=PostingsFormat(name=Memory), index_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), test_dt=PostingsFormat(name=Memory), out_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), whitetok=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{expr_s=DocValuesFormat(name=Lucene70), name_s=DocValuesFormat(name=Memory), iteration_i=DocValuesFormat(name=Memory), terms_ss=DocValuesFormat(name=Lucene70), join2_s=DocValuesFormat(name=Lucene70), field_s=DocValuesFormat(name=Direct), multiDefault=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Direct), a_f=DocValuesFormat(name=Direct), join1_i=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Lucene70), falseNegative_i=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Memory), join3_i=DocValuesFormat(name=Memory), test_i=DocValuesFormat(name=Lucene70), falsePositive_i=DocValuesFormat(name=Direct), field_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Lucene70), s_multi=DocValuesFormat(name=Lucene70), col_s=DocValuesFormat(name=Direct), a_ss=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Direct), a_s=DocValuesFormat(name=Lucene70), score_f=DocValuesFormat(name=Lucene70), term_s=DocValuesFormat(name=Lucene70), text_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Lucene70), i_multi=DocValuesFormat(name=Memory), featureSet_s=DocValuesFormat(name=Direct), trueNegative_i=DocValuesFormat(name=Lucene70), index_i=DocValuesFormat(name=Direct), b_ls=DocValuesFormat(name=Direct), test_dt=DocValuesFormat(name=Lucene70), out_i=DocValuesFormat(name=Direct), checkpoint_ss=DocValuesFormat(name=Memory), ident_s=DocValuesFormat(name=Direct), order_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Direct), side_s=DocValuesFormat(name=Memory), truePositive_i=DocValuesFormat(name=Lucene70), miles_i=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=986, maxMBSortInHeap=7.931305942845578, sim=RandomSimilarity(queryNorm=false): {}, locale=ru-RU, timezone=Europe/Simferopol
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=159314008,total=464162816
   [junit4]   2> NOTE: All tests run in this JVM: [ArcTangentEvaluatorTest, ModifiableSolrParamsTest, FloorEvaluatorTest, HttpSolrClientSSLAuthConPoolTest, StreamingTest, TestV1toV2ApiMapper, ConcatOperationTest, ArcCosineEvaluatorTest, HyperbolicSineEvaluatorTest, SchemaTest, TestSolrProperties, TangentEvaluatorTest, TestDelegationTokenRequest, NoOpResponseParserTest, TestDocumentObjectBinder, SolrDocumentTest, NamedListTest, TestBatchUpdate, LessThanEvaluatorTest, GraphTest, TestUpdateRequestCodec, DivideEvaluatorTest, FieldAnalysisResponseTest, SelectWithEvaluatorsTest, CloudSolrClientMultiConstructorTest, TestEmbeddedSolrServer, TestPathTrie, SolrExampleStreamingTest, StreamExpressionToExpessionTest, TestCloudSolrClientConnections, CloudSolrClientBuilderTest, TestJavaBinCodec, SolrExampleXMLTest, NotEvaluatorTest, SolrExampleStreamingBinaryTest, SolrExampleBinaryTest, LargeVolumeBinaryJettyTest, JettyWebappTest, LargeVolumeEmbeddedTest, TestSpellCheckResponse, LargeVolumeJettyTest, TermsResponseTest, QueryResponseTest, TestHash, ContentStreamTest, TestFastInputStream, TestXMLEscaping, AnlysisResponseBaseTest, FacetFieldTest, CollectionAdminRequestRequiredParamsTest, GetByIdTest, SolrSchemalessExampleTest, TestSolrJErrorHandling, CloudSolrClientCacheTest, CloudSolrClientTest, JDBCStreamTest, StreamExpressionTest]
   [junit4] Completed [135/135 (1!)] on J0 in 109.86s, 89 tests, 1 error <<< FAILURES!

[...truncated 42444 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4088 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4088/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

9 tests failed.
FAILED:  org.apache.solr.cloud.ReplaceNodeTest.test

Error Message:
org/apache/solr/client/solrj/request/CollectionAdminRequest$ReplaceNode

Stack Trace:
java.lang.NoClassDefFoundError: org/apache/solr/client/solrj/request/CollectionAdminRequest$ReplaceNode
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:FB21C485707DD75D]:0)
        at org.apache.solr.cloud.ReplaceNodeTest.test(ReplaceNodeTest.java:83)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.client.solrj.request.CollectionAdminRequest$ReplaceNode
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 40 more
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/classes/java/org/apache/solr/client/solrj/request/CollectionAdminRequest$ReplaceNode.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 46 more


FAILED:  org.apache.solr.cloud.TestRebalanceLeaders.test

Error Message:
timeout waiting for collection in cluster state: collection=testcollection

Stack Trace:
java.lang.RuntimeException: timeout waiting for collection in cluster state: collection=testcollection
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:FB21C485707DD75D]:0)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForCollection(AbstractFullDistribZkTestBase.java:361)
        at org.apache.solr.cloud.TestRebalanceLeaders.test(TestRebalanceLeaders.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:F5B403B281D06C45]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrCore(TestTlogReplica.java:849)
        at org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates(TestTlogReplica.java:652)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testBasicLeaderElection

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:E5FD3D17361BFD5F]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrRunner(TestTlogReplica.java:884)
        at org.apache.solr.cloud.TestTlogReplica.testBasicLeaderElection(TestTlogReplica.java:635)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionSplitShard

Error Message:
Error from server at http://127.0.0.1:51935/solr: ADDREPLICA failed to create replica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51935/solr: ADDREPLICA failed to create replica
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:A6FB88A46ECB3B1B]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1135)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:876)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionSplitShard(TestPolicyCloud.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddReplica

Error Message:
Error from server at http://127.0.0.1:51935/solr: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51935/solr: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:F3559E71CFC25203]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1135)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:876)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddReplica(TestPolicyCloud.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddShardUsingPolicy

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:51941/solr]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:51941/solr]
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:B2E9C9F61FB94766]:0)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:438)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1135)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:876)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddShardUsingPolicy(TestPolicyCloud.java:139)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:51941/solr
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:609)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:231)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:220)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        ... 45 more
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.getImpl(Socket.java:520)
        at java.net.Socket.setSoTimeout(Socket.java:1141)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:119)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:498)
        ... 49 more


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testDataProvider

Error Message:
Could not find collection : policiesTest

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : policiesTest
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:4BDE536A6C43BE7C]:0)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:194)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
        at org.apache.solr.cloud.autoscaling.TestPolicyCloud.testDataProvider(TestPolicyCloud.java:160)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.snapshots.TestSolrCloudSnapshots.testSnapshots

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
        at __randomizedtesting.SeedInfo.seed([7375FB5FDE81BAA5:45397FC0ABD8B8EA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.core.snapshots.TestSolrCloudSnapshots.testSnapshots(TestSolrCloudSnapshots.java:189)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12538 lines...]
   [junit4] Suite: org.apache.solr.cloud.ReplaceNodeTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/init-core-data-001
   [junit4]   2> 2008878 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2008884 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2008885 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 6 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001
   [junit4]   2> 2008885 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2008886 INFO  (Thread-4237) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2008886 INFO  (Thread-4237) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2008891 ERROR (Thread-4237) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2008996 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:50776
   [junit4]   2> 2009039 INFO  (jetty-launcher-4138-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2009039 INFO  (jetty-launcher-4138-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2009039 INFO  (jetty-launcher-4138-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2009040 INFO  (jetty-launcher-4138-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2009040 INFO  (jetty-launcher-4138-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2009048 INFO  (jetty-launcher-4138-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2029fa37{/solr,null,AVAILABLE}
   [junit4]   2> 2009050 INFO  (jetty-launcher-4138-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4bb1d9c0{HTTP/1.1,[http/1.1]}{127.0.0.1:50778}
   [junit4]   2> 2009050 INFO  (jetty-launcher-4138-thread-3) [    ] o.e.j.s.Server Started @2012241ms
   [junit4]   2> 2009050 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50778}
   [junit4]   2> 2009051 ERROR (jetty-launcher-4138-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2009051 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2009051 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2009051 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2009051 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T17:28:11.621Z
   [junit4]   2> 2009051 INFO  (jetty-launcher-4138-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@266a8a2f{/solr,null,AVAILABLE}
   [junit4]   2> 2009052 INFO  (jetty-launcher-4138-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b08ee87{/solr,null,AVAILABLE}
   [junit4]   2> 2009052 INFO  (jetty-launcher-4138-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@75254b0a{HTTP/1.1,[http/1.1]}{127.0.0.1:50780}
   [junit4]   2> 2009052 INFO  (jetty-launcher-4138-thread-5) [    ] o.e.j.s.Server Started @2012243ms
   [junit4]   2> 2009052 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50780}
   [junit4]   2> 2009053 ERROR (jetty-launcher-4138-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2009053 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64e6f792{/solr,null,AVAILABLE}
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T17:28:11.626Z
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6b3dac59{HTTP/1.1,[http/1.1]}{127.0.0.1:50782}
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-4) [    ] o.e.j.s.Server Started @2012247ms
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50782}
   [junit4]   2> 2009056 ERROR (jetty-launcher-4138-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2009056 INFO  (jetty-launcher-4138-thread-6) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2009055 INFO  (jetty-launcher-4138-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4389acba{/solr,null,AVAILABLE}
   [junit4]   2> 2009054 INFO  (jetty-launcher-4138-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@52bddd7e{HTTP/1.1,[http/1.1]}{127.0.0.1:50779}
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-1) [    ] o.e.j.s.Server Started @2012247ms
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50779}
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T17:28:11.627Z
   [junit4]   2> 2009057 ERROR (jetty-launcher-4138-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2009057 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2009058 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T17:28:11.628Z
   [junit4]   2> 2009059 INFO  (jetty-launcher-4138-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@13747698{HTTP/1.1,[http/1.1]}{127.0.0.1:50783}
   [junit4]   2> 2009059 INFO  (jetty-launcher-4138-thread-2) [    ] o.e.j.s.Server Started @2012250ms
   [junit4]   2> 2009059 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50783}
   [junit4]   2> 2009060 ERROR (jetty-launcher-4138-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2009060 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2009060 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2009060 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2009061 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T17:28:11.631Z
   [junit4]   2> 2009062 INFO  (jetty-launcher-4138-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2b390e70{/solr,null,AVAILABLE}
   [junit4]   2> 2009062 INFO  (jetty-launcher-4138-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7bce7f51{HTTP/1.1,[http/1.1]}{127.0.0.1:50787}
   [junit4]   2> 2009062 INFO  (jetty-launcher-4138-thread-6) [    ] o.e.j.s.Server Started @2012253ms
   [junit4]   2> 2009062 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50787}
   [junit4]   2> 2009063 ERROR (jetty-launcher-4138-thread-6) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2009063 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2009063 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2009063 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2009063 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T17:28:11.633Z
   [junit4]   2> 2009067 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2009068 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2009068 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2009068 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2009069 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2009069 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2009081 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009081 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009082 INFO  (jetty-launcher-4138-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50776/solr
   [junit4]   2> 2009083 INFO  (jetty-launcher-4138-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50776/solr
   [junit4]   2> 2009084 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30003, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009084 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30006, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009086 INFO  (jetty-launcher-4138-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50776/solr
   [junit4]   2> 2009087 INFO  (jetty-launcher-4138-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50776/solr
   [junit4]   2> 2009087 INFO  (jetty-launcher-4138-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50776/solr
   [junit4]   2> 2009088 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009089 INFO  (jetty-launcher-4138-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50776/solr
   [junit4]   2> 2009093 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30007, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009093 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30008, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009094 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c30009, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009094 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c3000a, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009095 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c3000b, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009097 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15cc165c5c3000c, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2009155 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2009155 INFO  (jetty-launcher-4138-thread-1) [n:127.0.0.1:50779_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2009156 INFO  (jetty-launcher-4138-thread-6) [n:127.0.0.1:50787_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2009156 INFO  (jetty-launcher-4138-thread-2) [n:127.0.0.1:50783_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2009156 INFO  (jetty-launcher-4138-thread-3) [n:127.0.0.1:50778_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2009156 INFO  (jetty-launcher-4138-thread-4) [n:127.0.0.1:50782_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2009157 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50780_solr
   [junit4]   2> 2009159 INFO  (jetty-launcher-4138-thread-1) [n:127.0.0.1:50779_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50779_solr
   [junit4]   2> 2009159 INFO  (jetty-launcher-4138-thread-6) [n:127.0.0.1:50787_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50787_solr
   [junit4]   2> 2009159 INFO  (jetty-launcher-4138-thread-2) [n:127.0.0.1:50783_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50783_solr
   [junit4]   2> 2009159 INFO  (jetty-launcher-4138-thread-3) [n:127.0.0.1:50778_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50778_solr
   [junit4]   2> 2009159 INFO  (jetty-launcher-4138-thread-4) [n:127.0.0.1:50782_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50782_solr
   [junit4]   2> 2009161 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.c.Overseer Overseer (id=98165934749057041-127.0.0.1:50780_solr-n_0000000000) starting
   [junit4]   2> 2009163 INFO  (zkCallback-4167-thread-1-processing-n:127.0.0.1:50782_solr) [n:127.0.0.1:50782_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2009163 INFO  (zkCallback-4168-thread-1-processing-n:127.0.0.1:50787_solr) [n:127.0.0.1:50787_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2009163 INFO  (zkCallback-4166-thread-1-processing-n:127.0.0.1:50779_solr) [n:127.0.0.1:50779_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2009164 INFO  (zkCallback-4165-thread-1-processing-n:127.0.0.1:50778_solr) [n:127.0.0.1:50778_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2009164 INFO  (zkCallback-4169-thread-1-processing-n:127.0.0.1:50780_solr) [n:127.0.0.1:50780_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2009164 INFO  (zkCallback-4170-thread-1-processing-n:127.0.0.1:50783_solr) [n:127.0.0.1:50783_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2009257 INFO  (jetty-launcher-4138-thread-1) [n:127.0.0.1:50779_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50779.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009261 INFO  (jetty-launcher-4138-thread-1) [n:127.0.0.1:50779_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50779.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009261 INFO  (jetty-launcher-4138-thread-1) [n:127.0.0.1:50779_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50779.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009262 INFO  (jetty-launcher-4138-thread-1) [n:127.0.0.1:50779_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node1/.
   [junit4]   2> 2009262 INFO  (jetty-launcher-4138-thread-3) [n:127.0.0.1:50778_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50778.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009265 INFO  (jetty-launcher-4138-thread-3) [n:127.0.0.1:50778_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50778.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009265 INFO  (jetty-launcher-4138-thread-3) [n:127.0.0.1:50778_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50778.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009266 INFO  (jetty-launcher-4138-thread-3) [n:127.0.0.1:50778_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node3/.
   [junit4]   2> 2009269 INFO  (jetty-launcher-4138-thread-2) [n:127.0.0.1:50783_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50783.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009272 INFO  (jetty-launcher-4138-thread-2) [n:127.0.0.1:50783_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50783.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009272 INFO  (jetty-launcher-4138-thread-2) [n:127.0.0.1:50783_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50783.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009273 INFO  (jetty-launcher-4138-thread-2) [n:127.0.0.1:50783_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node2/.
   [junit4]   2> 2009288 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50780_solr
   [junit4]   2> 2009292 INFO  (zkCallback-4167-thread-1-processing-n:127.0.0.1:50782_solr) [n:127.0.0.1:50782_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2009292 INFO  (zkCallback-4168-thread-1-processing-n:127.0.0.1:50787_solr) [n:127.0.0.1:50787_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2009292 INFO  (zkCallback-4165-thread-1-processing-n:127.0.0.1:50778_solr) [n:127.0.0.1:50778_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2009292 INFO  (zkCallback-4166-thread-1-processing-n:127.0.0.1:50779_solr) [n:127.0.0.1:50779_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2009292 INFO  (zkCallback-4169-thread-1-processing-n:127.0.0.1:50780_solr) [n:127.0.0.1:50780_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2009293 INFO  (zkCallback-4170-thread-1-processing-n:127.0.0.1:50783_solr) [n:127.0.0.1:50783_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2009320 INFO  (jetty-launcher-4138-thread-6) [n:127.0.0.1:50787_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50787.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009324 INFO  (jetty-launcher-4138-thread-6) [n:127.0.0.1:50787_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50787.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009324 INFO  (jetty-launcher-4138-thread-6) [n:127.0.0.1:50787_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50787.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009325 INFO  (jetty-launcher-4138-thread-6) [n:127.0.0.1:50787_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node6/.
   [junit4]   2> 2009353 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50780.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009357 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50780.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009357 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50780.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009358 INFO  (jetty-launcher-4138-thread-5) [n:127.0.0.1:50780_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node5/.
   [junit4]   2> 2009413 INFO  (jetty-launcher-4138-thread-4) [n:127.0.0.1:50782_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50782.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009415 INFO  (jetty-launcher-4138-thread-4) [n:127.0.0.1:50782_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50782.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009415 INFO  (jetty-launcher-4138-thread-4) [n:127.0.0.1:50782_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50782.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2009416 INFO  (jetty-launcher-4138-thread-4) [n:127.0.0.1:50782_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node4/.
   [junit4]   2> 2009451 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 2009452 INFO  (SUITE-ReplaceNodeTest-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50776/solr ready
   [junit4]   2> 2009481 INFO  (TEST-ReplaceNodeTest.test-seed#[7375FB5FDE81BAA5]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2009484 INFO  (TEST-ReplaceNodeTest.test-seed#[7375FB5FDE81BAA5]) [    ] o.a.s.c.ReplaceNodeTest total_jettys: 6
   [junit4]   2> 2009485 INFO  (qtp1516675660-21209) [n:127.0.0.1:50783_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=3&name=replacenodetest_coll&nrtReplicas=1&action=CREATE&numShards=5&tlogReplicas=1&createNodeSet=127.0.0.1:50779_solr,127.0.0.1:50782_solr,127.0.0.1:50778_solr,127.0.0.1:50780_solr,127.0.0.1:50783_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2009488 INFO  (OverseerThreadFactory-9508-thread-1-processing-n:127.0.0.1:50780_solr) [n:127.0.0.1:50780_solr    ] o.a.s.c.CreateCollectionCmd Create collection replacenodetest_coll
   [junit4]   2> 2009605 INFO  (qtp959638424-21201) [n:127.0.0.1:50778_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard5_replica_t1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard5&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2009606 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard3_replica_t1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard3&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2009606 INFO  (qtp959638424-21201) [n:127.0.0.1:50778_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2009606 INFO  (qtp1308534890-21224) [n:127.0.0.1:50780_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard2_replica_n1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2009607 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2009607 INFO  (qtp1516675660-21210) [n:127.0.0.1:50783_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard4_replica_n1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard4&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2009607 INFO  (qtp1308534890-21224) [n:127.0.0.1:50780_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2009608 INFO  (qtp1516675660-21210) [n:127.0.0.1:50783_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2009608 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard1_replica_t1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2009609 INFO  (qtp596369793-21217) [n:127.0.0.1:50779_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard5_replica_n1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard5&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2009610 INFO  (qtp596369793-21217) [n:127.0.0.1:50779_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2009610 INFO  (qtp1308534890-21227) [n:127.0.0.1:50780_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard4_replica_t1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard4&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2009610 INFO  (qtp1583399407-21236) [n:127.0.0.1:50782_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard1_replica_n1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2009611 INFO  (qtp596369793-21218) [n:127.0.0.1:50779_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard2_replica_t1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2009611 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_shard3_replica_n1&action=CREATE&numShards=5&collection=replacenodetest_coll&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2009763 INFO  (zkCallback-4167-thread-1-processing-n:127.0.0.1:50782_solr) [n:127.0.0.1:50782_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/replacenodetest_coll/state.json] for collection [replacenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 2009763 INFO  (zkCallback-4170-thread-1-processing-n:127.0.0.1:50783_solr) [n:127.0.0.1:50783_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/replacenodetest_coll/state.json] for collection [replacenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 2009763 INFO  (zkCallback-4166-thread-1-processing-n:127.0.0.1:50779_solr) [n:127.0.0.1:50779_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/replacenodetest_coll/state.json] for collection [replacenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 2009763 INFO  (zkCallback-4165-thread-1-processing-n:127.0.0.1:50778_solr) [n:127.0.0.1:50778_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/replacenodetest_coll/state.json] for collection [replacenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 2009763 INFO  (zkCallback-4169-thread-1-processing-n:127.0.0.1:50780_solr) [n:127.0.0.1:50780_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/replacenodetest_coll/state.json] for collection [replacenodetest_coll] has occurred - updating... (live nodes size: [6])
   [junit4]   2> 2010647 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010648 INFO  (qtp959638424-21201) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard5  x:replacenodetest_coll_shard5_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010649 INFO  (qtp1516675660-21210) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard4  x:replacenodetest_coll_shard4_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010657 INFO  (qtp959638424-21201) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard5  x:replacenodetest_coll_shard5_replica_t1] o.a.s.s.IndexSchema [replacenodetest_coll_shard5_replica_t1] Schema name=cloud-dynamic
   [junit4]   2> 2010657 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.s.IndexSchema [replacenodetest_coll_shard3_replica_t1] Schema name=cloud-dynamic
   [junit4]   2> 2010658 INFO  (qtp1516675660-21210) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard4  x:replacenodetest_coll_shard4_replica_n1] o.a.s.s.IndexSchema [replacenodetest_coll_shard4_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2010660 INFO  (qtp1308534890-21224) [n:127.0.0.1:50780_solr c:replacenodetest_coll s:shard2  x:replacenodetest_coll_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010660 INFO  (qtp596369793-21217) [n:127.0.0.1:50779_solr c:replacenodetest_coll s:shard5  x:replacenodetest_coll_shard5_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010661 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010674 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010674 INFO  (qtp596369793-21218) [n:127.0.0.1:50779_solr c:replacenodetest_coll s:shard2  x:replacenodetest_coll_shard2_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010676 INFO  (qtp1583399407-21236) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010674 INFO  (qtp1308534890-21227) [n:127.0.0.1:50780_solr c:replacenodetest_coll s:shard4  x:replacenodetest_coll_shard4_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2010682 INFO  (qtp596369793-21218) [n:127.0.0.1:50779_solr c:replacenodetest_coll s:shard2  x:replacenodetest_coll_shard2_replica_t1] o.a.s.s.IndexSchema [replacenodetest_coll_shard2_replica_t1] Schema name=cloud-dynamic
   [junit4]   2> 2010682 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.s.IndexSchema [replacenodetest_coll_shard3_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2010682 INFO  (qtp1583399407-21236) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_n1] o.a.s.s.IndexSchema [replacenodetest_coll_shard1_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2010700 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.s.IndexSchema [replacenodetest_coll_shard1_replica_t1] Schema name=cloud-dynamic
   [junit4]   2> 2010701 INFO  (qtp596369793-21217) [n:127.0.0.1:50779_solr c:replacenodetest_coll s:shard5  x:replacenodetest_coll_shard5_replica_n1] o.a.s.s.IndexSchema [replacenodetest_coll_shard5_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2010703 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2010703 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'replacenodetest_coll_shard3_replica_n1' using configuration from collection replacenodetest_coll, trusted=true
   [junit4]   2> 2010710 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50778.solr.core.replacenodetest_coll.shard3.replica_n1' (registry 'solr.core.replacenodetest_coll.shard3.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2010710 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2010710 INFO  (qtp959638424-21202) [n:127.0.0.1:50778_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_n1] o.a.s.c.SolrCore [[replacenodetest_coll_shard3_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node3/replacenodetest_coll_shard3_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node3/./replacenodetest_coll_shard3_replica_n1/data/]
   [junit4]   2> 2010714 INFO  (qtp1308534890-21227) [n:127.0.0.1:50780_solr c:replacenodetest_coll s:shard4  x:replacenodetest_coll_shard4_replica_t1] o.a.s.s.IndexSchema [replacenodetest_coll_shard4_replica_t1] Schema name=cloud-dynamic
   [junit4]   2> 2010719 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2010719 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'replacenodetest_coll_shard1_replica_t1' using configuration from collection replacenodetest_coll, trusted=true
   [junit4]   2> 2010719 INFO  (qtp1308534890-21224) [n:127.0.0.1:50780_solr c:replacenodetest_coll s:shard2  x:replacenodetest_coll_shard2_replica_n1] o.a.s.s.IndexSchema [replacenodetest_coll_shard2_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 2010719 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50783.solr.core.replacenodetest_coll.shard1.replica_t1' (registry 'solr.core.replacenodetest_coll.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2010719 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2010719 INFO  (qtp1516675660-21211) [n:127.0.0.1:50783_solr c:replacenodetest_coll s:shard1  x:replacenodetest_coll_shard1_replica_t1] o.a.s.c.SolrCore [[replacenodetest_coll_shard1_replica_t1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node2/replacenodetest_coll_shard1_replica_t1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ReplaceNodeTest_7375FB5FDE81BAA5-001/tempDir-001/node2/./replacenodetest_coll_shard1_replica_t1/data/]
   [junit4]   2> 2010737 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2010737 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'replacenodetest_coll_shard3_replica_t1' using configuration from collection replacenodetest_coll, trusted=true
   [junit4]   2> 2010738 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50782.solr.core.replacenodetest_coll.shard3.replica_t1' (registry 'solr.core.replacenodetest_coll.shard3.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a917a8c
   [junit4]   2> 2010738 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2010739 INFO  (qtp1583399407-21226) [n:127.0.0.1:50782_solr c:replacenodetest_coll s:shard3  x:replacenodetest_coll_shard3_replica_t1] o.a.s.c.SolrCore

[...truncated too long message...]

ueue Response ZK path: /overseer/collection-queue-work/qnr-0000000005 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2691579 WARN  (zkCallback-4665-thread-3-processing-n:127.0.0.1:52094_solr) [n:127.0.0.1:52094_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2691580 INFO  (jetty-closer-4644-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7c87d7e8{/solr,null,UNAVAILABLE}
   [junit4]   2> 2691582 ERROR (SUITE-TestSolrCloudSnapshots-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2691583 INFO  (SUITE-TestSolrCloudSnapshots-seed#[7375FB5FDE81BAA5]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52087 52087
   [junit4]   2> 2691605 INFO  (Thread-4504) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52087 52087
   [junit4]   2> 2691840 WARN  (Thread-4504) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/aliases.json
   [junit4]   2> 5 /solr/clusterprops.json
   [junit4]   2> 4 /solr/security.json
   [junit4]   2> 4 /solr/configs/conf1
   [junit4]   2> 3 /solr/collections/SolrCloudSnapshots_restored/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/clusterstate.json
   [junit4]   2> 4 /solr/collections/SolrCloudSnapshots/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2> 5 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.snapshots.TestSolrCloudSnapshots_7375FB5FDE81BAA5-001
   [junit4]   2> Jun 19, 2017 5:39:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=sr-Latn, timezone=America/Marigot
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=151700464,total=521666560
   [junit4]   2> NOTE: All tests run in this JVM: [TestSubQueryTransformerDistrib, FileBasedSpellCheckerTest, HdfsBasicDistributedZkTest, TestClassicSimilarityFactory, TestNumericTerms32, SpatialHeatmapFacetsTest, TestDocumentBuilder, CircularListTest, TestEmbeddedSolrServerConstructors, TestNoOpRegenerator, TestSha256AuthenticationProvider, CheckHdfsIndexTest, CdcrReplicationDistributedZkTest, TestFieldResource, TestReplicationHandlerBackup, TestAtomicUpdateErrorCases, TestDynamicLoading, TestQuerySenderListener, TestGeoJSONResponseWriter, DeleteLastCustomShardedReplicaTest, TestSolrCloudWithKerberosAlt, TestRandomFaceting, BasicFunctionalityTest, FieldMutatingUpdateProcessorTest, TestLMJelinekMercerSimilarityFactory, MetricUtilsTest, HttpPartitionTest, SolrMetricReporterTest, TestShardHandlerFactory, TestHighFrequencyDictionaryFactory, TestManagedResourceStorage, MetricsHandlerTest, TestReversedWildcardFilterFactory, TestSolrConfigHandlerConcurrent, TestExceedMaxTermLength, CoreAdminCreateDiscoverTest, PreAnalyzedUpdateProcessorTest, TestDocSet, TestDistributedMissingSort, TestSolrConfigHandler, TestFieldTypeResource, TestAuthenticationFramework, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestUninvertingReader, TlogReplayBufferedWhileIndexingTest, JavabinLoaderTest, OverseerModifyCollectionTest, LoggingHandlerTest, TestLeaderInitiatedRecoveryThread, CachingDirectoryFactoryTest, TestSchemaResource, TestSolrCoreProperties, BitVectorTest, TestQueryTypes, TestUnifiedSolrHighlighter, CloneFieldUpdateProcessorFactoryTest, TestTrie, SolrJmxReporterTest, TestLRUCache, BJQParserTest, TestFileDictionaryLookup, NotRequiredUniqueKeyTest, TestAnalyzedSuggestions, SolrGangliaReporterTest, TestCollectionAPIs, TestSolrQueryResponse, DefaultValueUpdateProcessorTest, RuleEngineTest, FieldAnalysisRequestHandlerTest, PKIAuthenticationIntegrationTest, DistributedIntervalFacetingTest, TestCustomDocTransformer, TestInPlaceUpdatesDistrib, TestImpersonationWithHadoopAuth, LargeFieldTest, DirectUpdateHandlerOptimizeTest, TestSchemaVersionResource, JvmMetricsTest, TermVectorComponentDistributedTest, TestFieldCacheSortRandom, TestCloudManagedSchema, RemoteQueryErrorTest, CdcrRequestHandlerTest, TestCustomSort, TestExpandComponent, BasicAuthIntegrationTest, DeleteReplicaTest, TestIndexSearcher, TestNonDefinedSimilarityFactory, SuggesterTest, TimeZoneUtilsTest, TestFiltering, DistributedFacetPivotSmallAdvancedTest, TestStressCloudBlindAtomicUpdates, SpatialRPTFieldTypeTest, TestFastWriter, EchoParamsTest, TestLegacyTerms, DeleteStatusTest, TestBulkSchemaAPI, DistribDocExpirationUpdateProcessorTest, HdfsChaosMonkeySafeLeaderTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, CloudMLTQParserTest, DocValuesMissingTest, TestCollectionAPI, TestStandardQParsers, TestCrossCoreJoin, UtilsToolTest, TestCryptoKeys, CustomCollectionTest, LeaderInitiatedRecoveryOnCommitTest, TestHashQParserPlugin, TestPHPSerializedResponseWriter, CoreSorterTest, TestDistribDocBasedVersion, ReturnFieldsTest, TestRawResponseWriter, ConnectionManagerTest, TestFieldCacheReopen, LeaderElectionIntegrationTest, TestPKIAuthenticationPlugin, TestStressLiveNodes, TestConfigSetsAPIExclusivity, ResponseLogComponentTest, TestCloudRecovery, TestRecoveryHdfs, TestCopyFieldCollectionResource, TestDistribIDF, TestReplicaProperties, TestDistributedGrouping, BlockJoinFacetRandomTest, DOMUtilTest, TestCloudSchemaless, ScriptEngineTest, TestSolrCLIRunExample, CloudExitableDirectoryReaderTest, AssignTest, HdfsDirectoryTest, TestQueryUtils, TestComplexPhraseLeadingWildcard, MigrateRouteKeyTest, TermVectorComponentTest, WordBreakSolrSpellCheckerTest, OverseerStatusTest, TestCodecSupport, OverriddenZkACLAndCredentialsProvidersTest, ReplicationFactorTest, TestNumericRangeQuery32, TestJettySolrRunner, HdfsWriteToMultipleCollectionsTest, DistanceUnitsTest, TestComponentsName, ConcurrentDeleteAndCreateCollectionTest, DocValuesNotIndexedTest, SpellCheckCollatorTest, TestPivotHelperCode, TestUseDocValuesAsStored2, TestCloudPivotFacet, AnalyticsQueryTest, TestRestoreCore, TestSubQueryTransformer, SolrInfoBeanTest, TestLegacyFieldReuse, TestScoreJoinQPScore, TestStressRecovery, GraphQueryTest, TestFilteredDocIdSet, TestJavabinTupleStreamParser, TemplateUpdateProcessorTest, TestInfoStreamLogging, RequestHandlersTest, BasicDistributedZk2Test, CSVRequestHandlerTest, TestSolrDeletionPolicy2, SignatureUpdateProcessorFactoryTest, QueryParsingTest, TestMissingGroups, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, ShardRoutingCustomTest, TestRecovery, TestHashPartitioner, TestReload, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, StatsComponentTest, SolrCmdDistributorTest, BadIndexSchemaTest, ConvertedLegacyTest, DirectUpdateHandlerTest, TestBadConfig, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, CoreAdminHandlerTest, SuggesterTSTTest, TestStressLucene, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, SortByFunctionTest, DistanceFunctionTest, DebugComponentTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, TestIndexingPerformance, FastVectorHighlighterTest, IndexSchemaTest, TestCSVResponseWriter, JsonLoaderTest, PingRequestHandlerTest, HighlighterConfigTest, SOLR749Test, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, SolrIndexConfigTest, SampleTest, TestConfig, OutputWriterTest, ExternalFileFieldSortTest, DirectSolrConnectionTest, TestPhraseSuggestions, TestDFRSimilarityFactory, TestIBSimilarityFactory, OpenExchangeRatesOrgProviderTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, CleanupOldIndexTest, CollectionStateFormat2Test, CollectionsAPIAsyncDistributedZkTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DistribJoinFromCollectionTest, ForceLeaderTest, PeerSyncReplicationTest, RecoveryAfterSoftCommitTest, RollingRestartTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestDeleteCollectionOnDownNodes, TestExclusionRuleCollectionAccess, TestLeaderElectionZkExpiry, TestLockTree, TestOnReconnectListenerSupport, TestRandomRequestDistribution, TestRequestStatusCollectionAPI, TestSegmentSorting, TestShortCircuitedRequests, TestStressInPlaceUpdates, TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, VMParamsZkACLAndCredentialsProvidersTest, AutoScalingHandlerTest, TestPolicyCloud, HdfsChaosMonkeyNothingIsSafeTest, HdfsCollectionsAPIDistributedZkTest, HdfsNNFailoverTest, HdfsRecoveryZkTest, HdfsSyncSliceTest, HdfsThreadLeakTest, HdfsTlogReplayBufferedWhileIndexingTest, TestClusterStateMutator, ZkStateWriterTest, ImplicitSnitchTest, BlobRepositoryCloudTest, DirectoryFactoryTest, HdfsDirectoryFactoryTest, QueryResultKeyTest, TestConfigSetProperties, TestCorePropertiesReload, TestInitParams, TestNRTOpen, TestReloadAndDeleteDocs, TestSimpleTextCodec, TestSolrCloudSnapshots]
   [junit4] Completed [675/725 (5!)] on J1 in 342.98s, 1 test, 1 failure <<< FAILURES!

[...truncated 43453 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4089 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4089/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded  at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)  at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)  at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748) ,time=1}

Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
        at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
        at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
,time=1}
        at __randomizedtesting.SeedInfo.seed([3918DC1B6351AFA1:B14CE3C1CDADC259]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1186)
        at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1127)
        at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:987)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1011)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.TestDynamicLoading.testDynamicLoading

Error Message:
Could not get expected value  'org.apache.solr.core.BlobStoreTestRequestHandler' for path 'overlay/requestHandler/\/test1/class' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "overlay":{     "znodeVersion":0,     "runtimeLib":{"colltest":{         "name":"colltest",         "version":1}}}},  from server:  null

Stack Trace:
java.lang.AssertionError: Could not get expected value  'org.apache.solr.core.BlobStoreTestRequestHandler' for path 'overlay/requestHandler/\/test1/class' full output: {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "overlay":{
    "znodeVersion":0,
    "runtimeLib":{"colltest":{
        "name":"colltest",
        "version":1}}}},  from server:  null
        at __randomizedtesting.SeedInfo.seed([3918DC1B6351AFA1:E155F14C948C0A01]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:556)
        at org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:97)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11106 lines...]
   [junit4] Suite: org.apache.solr.core.TestDynamicLoading
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/init-core-data-001
   [junit4]   2> 268914 WARN  (SUITE-TestDynamicLoading-seed#[3918DC1B6351AFA1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 268914 INFO  (SUITE-TestDynamicLoading-seed#[3918DC1B6351AFA1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 268917 INFO  (SUITE-TestDynamicLoading-seed#[3918DC1B6351AFA1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 268917 INFO  (SUITE-TestDynamicLoading-seed#[3918DC1B6351AFA1]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /gi_/x
   [junit4]   2> 268919 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 268919 INFO  (Thread-580) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 268919 INFO  (Thread-580) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 268922 ERROR (Thread-580) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 269029 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.ZkTestServer start zk server on port:54023
   [junit4]   2> 269073 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 269077 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 269079 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 269081 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 269083 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 269086 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 269088 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 269091 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 269093 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 269095 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 269098 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 269101 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 269361 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/control-001/cores/collection1
   [junit4]   2> 269365 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 269367 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31e7d906{/gi_/x,null,AVAILABLE}
   [junit4]   2> 269368 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7907d756{HTTP/1.1,[http/1.1]}{127.0.0.1:54026}
   [junit4]   2> 269368 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.Server Started @273347ms
   [junit4]   2> 269368 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/tempDir-001/control/data, replicaType=NRT, hostContext=/gi_/x, hostPort=54026, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/control-001/cores}
   [junit4]   2> 269368 ERROR (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 269369 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 269369 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 269369 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 269369 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-20T01:09:08.544Z
   [junit4]   2> 269373 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 269373 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/control-001/solr.xml
   [junit4]   2> 269384 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 269392 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54023/solr
   [junit4]   2> 269434 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 269436 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54026_gi_%2Fx
   [junit4]   2> 269438 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.Overseer Overseer (id=98167747254419460-127.0.0.1:54026_gi_%2Fx-n_0000000000) starting
   [junit4]   2> 269452 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54026_gi_%2Fx
   [junit4]   2> 269455 INFO  (zkCallback-460-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 269529 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 269535 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 269535 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 269537 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/control-001/cores
   [junit4]   2> 269537 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 269538 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 269541 INFO  (OverseerStateUpdate-98167747254419460-127.0.0.1:54026_gi_%2Fx-n_0000000000) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 270563 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 270578 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 270674 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 270696 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 270696 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.core_node1' (registry 'solr.core.control_collection.shard1.core_node1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 270696 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 270696 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/control-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/control-001/cores/collection1/data/]
   [junit4]   2> 270698 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=5.8896484375, floorSegmentMB=1.4140625, forceMergeDeletesPctAllowed=14.92553698740939, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 270700 WARN  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 270752 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 270752 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 270753 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 270753 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 270753 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=64.1650390625, floorSegmentMB=1.9306640625, forceMergeDeletesPctAllowed=5.99953385999077, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 270753 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@e433c9[collection1] main]
   [junit4]   2> 270757 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 270758 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 270758 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 270758 INFO  (coreLoadExecutor-1413-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570683957996945408
   [junit4]   2> 270759 INFO  (searcherExecutor-1414-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@e433c9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 270767 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 270767 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 270767 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54026/gi_/x/collection1/
   [junit4]   2> 270767 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 270767 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:54026/gi_/x/collection1/ has no replicas
   [junit4]   2> 270767 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 270772 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54026/gi_/x/collection1/ shard1
   [junit4]   2> 270946 INFO  (coreZkRegister-1406-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx x:collection1 c:control_collection) [n:127.0.0.1:54026_gi_%2Fx c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 271086 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 271088 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54023/solr ready
   [junit4]   2> 271088 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 271088 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 271330 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001/cores/collection1
   [junit4]   2> 271331 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001 of type NRT
   [junit4]   2> 271339 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 271341 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e22529d{/gi_/x,null,AVAILABLE}
   [junit4]   2> 271342 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@49c184a6{HTTP/1.1,[http/1.1]}{127.0.0.1:54032}
   [junit4]   2> 271342 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.e.j.s.Server Started @275321ms
   [junit4]   2> 271342 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/gi_/x, hostPort=54032, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001/cores}
   [junit4]   2> 271342 ERROR (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 271342 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 271342 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 271343 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 271343 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-20T01:09:10.518Z
   [junit4]   2> 271348 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 271348 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001/solr.xml
   [junit4]   2> 271353 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 271357 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54023/solr
   [junit4]   2> 271377 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 271381 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 271385 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54032_gi_%2Fx
   [junit4]   2> 271387 INFO  (zkCallback-464-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 271387 INFO  (zkCallback-470-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 271388 INFO  (zkCallback-460-thread-1-processing-n:127.0.0.1:54026_gi_%2Fx) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 271481 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 271484 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 271484 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 271486 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001/cores
   [junit4]   2> 271486 INFO  (TEST-TestDynamicLoading.testDynamicLoading-seed#[3918DC1B6351AFA1]) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 271487 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 271490 INFO  (OverseerStateUpdate-98167747254419460-127.0.0.1:54026_gi_%2Fx-n_0000000000) [n:127.0.0.1:54026_gi_%2Fx    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 271603 INFO  (zkCallback-470-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 272517 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 272532 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 272634 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 272653 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 272654 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.core_node1' (registry 'solr.core.collection1.shard2.core_node1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3119fc22
   [junit4]   2> 272654 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 272654 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_3918DC1B6351AFA1-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 272655 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=5.8896484375, floorSegmentMB=1.4140625, forceMergeDeletesPctAllowed=14.92553698740939, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 272658 WARN  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 272758 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 272758 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 272759 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 272759 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 272760 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=64.1650390625, floorSegmentMB=1.9306640625, forceMergeDeletesPctAllowed=5.99953385999077, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 272760 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1c5ccd31[collection1] main]
   [junit4]   2> 272762 INFO  (coreLoadExecutor-1424-thread-1-processing-n:127.0.0.1:54032_gi_%2Fx) [n:127.0.0.1:54032_gi_%2Fx c:collection1   x:collection1] o.a.s.r.ManagedResourceSt