[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4417 - Still unstable!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4417 - Still unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4417/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingSorting

Error Message:
Should have exactly 4 documents returned expected:<4> but was:<3>

Stack Trace:
java.lang.AssertionError: Should have exactly 4 documents returned expected:<4> but was:<3>
        at __randomizedtesting.SeedInfo.seed([2EB9A973366553B3:3081A17B4ACEE933]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.DocValuesNotIndexedTest.checkSortOrder(DocValuesNotIndexedTest.java:260)
        at org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingSorting(DocValuesNotIndexedTest.java:245)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([2EB9A973366553B3:94B59EFC698D85FC]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.junit.Assert.assertNull(Assert.java:562)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1575)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13309 lines...]
   [junit4] Suite: org.apache.solr.cloud.DocValuesNotIndexedTest
   [junit4]   2> 1958020 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/init-core-data-001
   [junit4]   2> 1958021 WARN  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=66 numCloses=66
   [junit4]   2> 1958021 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1958028 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1958028 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001
   [junit4]   2> 1958029 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1958029 INFO  (Thread-4025) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1958029 INFO  (Thread-4025) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1958032 ERROR (Thread-4025) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1958135 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:54170
   [junit4]   2> 1958145 INFO  (zkConnectionManagerCallback-4799-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958173 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1958173 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1958175 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1958175 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1958175 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1958177 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@49161772{/solr,null,AVAILABLE}
   [junit4]   2> 1958184 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1958184 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1958184 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1958206 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@113cdc92{/solr,null,AVAILABLE}
   [junit4]   2> 1958207 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@503d7894{HTTP/1.1,[http/1.1]}{127.0.0.1:54174}
   [junit4]   2> 1958208 INFO  (jetty-launcher-4796-thread-1) [    ] o.e.j.s.Server Started @1961732ms
   [junit4]   2> 1958208 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54174}
   [junit4]   2> 1958208 ERROR (jetty-launcher-4796-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1958209 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@27165837{HTTP/1.1,[http/1.1]}{127.0.0.1:54175}
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-2) [    ] o.e.j.s.Server Started @1961734ms
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54175}
   [junit4]   2> 1958210 ERROR (jetty-launcher-4796-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1958210 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1958211 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1958211 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-31T22:22:06.263Z
   [junit4]   2> 1958211 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-31T22:22:06.263Z
   [junit4]   2> 1958236 INFO  (zkConnectionManagerCallback-4801-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958238 INFO  (zkConnectionManagerCallback-4803-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958244 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1958244 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1958273 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10038b151af0001, likely client has closed socket
   [junit4]   2> 1958275 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10038b151af0002, likely client has closed socket
   [junit4]   2> 1958281 INFO  (jetty-launcher-4796-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54170/solr
   [junit4]   2> 1958281 INFO  (jetty-launcher-4796-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54170/solr
   [junit4]   2> 1958283 INFO  (zkConnectionManagerCallback-4811-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958284 INFO  (zkConnectionManagerCallback-4809-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958291 INFO  (zkConnectionManagerCallback-4815-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958292 INFO  (zkConnectionManagerCallback-4814-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1958606 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1958606 INFO  (jetty-launcher-4796-thread-1) [n:127.0.0.1:54174_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1958607 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54175_solr
   [junit4]   2> 1958608 INFO  (jetty-launcher-4796-thread-1) [n:127.0.0.1:54174_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54174_solr
   [junit4]   2> 1958609 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.c.Overseer Overseer (id=72119928268718086-127.0.0.1:54175_solr-n_0000000000) starting
   [junit4]   2> 1958610 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1958610 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1958653 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54175_solr
   [junit4]   2> 1958661 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1958664 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1959017 INFO  (jetty-launcher-4796-thread-1) [n:127.0.0.1:54174_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54174.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1959065 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1959094 INFO  (jetty-launcher-4796-thread-1) [n:127.0.0.1:54174_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54174.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1959095 INFO  (jetty-launcher-4796-thread-1) [n:127.0.0.1:54174_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54174.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1959105 INFO  (jetty-launcher-4796-thread-1) [n:127.0.0.1:54174_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/.
   [junit4]   2> 1959121 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1959121 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1959122 INFO  (jetty-launcher-4796-thread-2) [n:127.0.0.1:54175_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/.
   [junit4]   2> 1959339 INFO  (zkConnectionManagerCallback-4821-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1959343 INFO  (zkConnectionManagerCallback-4825-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1959345 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1959346 INFO  (SUITE-DocValuesNotIndexedTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54170/solr ready
   [junit4]   2> 1959749 INFO  (qtp897872790-19580) [n:127.0.0.1:54174_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&maxShardsPerNode=2&name=dv_coll&nrtReplicas=1&action=CREATE&numShards=4&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1959753 INFO  (OverseerThreadFactory-8622-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection dv_coll
   [junit4]   2> 1959869 INFO  (OverseerStateUpdate-72119928268718086-127.0.0.1:54175_solr-n_0000000000) [n:127.0.0.1:54175_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"dv_coll",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"dv_coll_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:54175/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1959873 INFO  (OverseerStateUpdate-72119928268718086-127.0.0.1:54175_solr-n_0000000000) [n:127.0.0.1:54175_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"dv_coll",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"dv_coll_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:54174/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1959877 INFO  (OverseerStateUpdate-72119928268718086-127.0.0.1:54175_solr-n_0000000000) [n:127.0.0.1:54175_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"dv_coll",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "core":"dv_coll_shard3_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:54175/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1959882 INFO  (OverseerStateUpdate-72119928268718086-127.0.0.1:54175_solr-n_0000000000) [n:127.0.0.1:54175_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"dv_coll",
   [junit4]   2>   "shard":"shard4",
   [junit4]   2>   "core":"dv_coll_shard4_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:54174/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1960102 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf1&newCollection=true&name=dv_coll_shard2_replica_n2&action=CREATE&numShards=4&collection=dv_coll&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1960103 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&newCollection=true&name=dv_coll_shard1_replica_n1&action=CREATE&numShards=4&collection=dv_coll&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1960102 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf1&newCollection=true&name=dv_coll_shard3_replica_n4&action=CREATE&numShards=4&collection=dv_coll&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1960103 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1960103 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1960104 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=dv_coll_shard4_replica_n6&action=CREATE&numShards=4&collection=dv_coll&shard=shard4&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1960474 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1960475 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1960475 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1960476 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1961461 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1961463 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1961467 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1961541 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1961545 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.s.IndexSchema [dv_coll_shard4_replica_n6] Schema name=minimal
   [junit4]   2> 1961548 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.IndexSchema [dv_coll_shard3_replica_n4] Schema name=minimal
   [junit4]   2> 1961548 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.IndexSchema [dv_coll_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 1961550 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1961550 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.IndexSchema [dv_coll_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1961550 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'dv_coll_shard3_replica_n4' using configuration from collection dv_coll, trusted=true
   [junit4]   2> 1961550 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1961550 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'dv_coll_shard4_replica_n6' using configuration from collection dv_coll, trusted=true
   [junit4]   2> 1961553 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.core.dv_coll.shard3.replica_n4' (registry 'solr.core.dv_coll.shard3.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1961554 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1961554 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore [[dv_coll_shard3_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/dv_coll_shard3_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/./dv_coll_shard3_replica_n4/data/]
   [junit4]   2> 1961553 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54174.solr.core.dv_coll.shard4.replica_n6' (registry 'solr.core.dv_coll.shard4.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1961554 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1961554 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SolrCore [[dv_coll_shard4_replica_n6] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/dv_coll_shard4_replica_n6], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/./dv_coll_shard4_replica_n6/data/]
   [junit4]   2> 1961555 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1961555 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'dv_coll_shard2_replica_n2' using configuration from collection dv_coll, trusted=true
   [junit4]   2> 1961555 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54174.solr.core.dv_coll.shard2.replica_n2' (registry 'solr.core.dv_coll.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1961555 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1961556 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore [[dv_coll_shard2_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/dv_coll_shard2_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/./dv_coll_shard2_replica_n2/data/]
   [junit4]   2> 1961559 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1961560 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'dv_coll_shard1_replica_n1' using configuration from collection dv_coll, trusted=true
   [junit4]   2> 1961560 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.core.dv_coll.shard1.replica_n1' (registry 'solr.core.dv_coll.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1961560 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1961560 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [[dv_coll_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/dv_coll_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/./dv_coll_shard1_replica_n1/data/]
   [junit4]   2> 1961657 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1961657 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1961661 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1961661 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1961667 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@4f7e603f[dv_coll_shard4_replica_n6] main]
   [junit4]   2> 1961689 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1961689 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1961692 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1961692 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1961715 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1e8a6ae[dv_coll_shard2_replica_n2] main]
   [junit4]   2> 1961721 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1961722 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1961722 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1961724 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1961724 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1961725 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148365508247552
   [junit4]   2> 1961725 INFO  (searcherExecutor-8629-thread-1-processing-n:127.0.0.1:54174_solr x:dv_coll_shard2_replica_n2 s:shard2 c:dv_coll r:core_node5) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore [dv_coll_shard2_replica_n2] Registered new searcher Searcher@1e8a6ae[dv_coll_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1961728 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1961729 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1961735 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1961735 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1961735 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54174/solr/dv_coll_shard2_replica_n2/
   [junit4]   2> 1961736 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1961736 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:54174/solr/dv_coll_shard2_replica_n2/ has no replicas
   [junit4]   2> 1961736 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1961736 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1961736 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1961741 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54174/solr/dv_coll_shard2_replica_n2/ shard2
   [junit4]   2> 1961743 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1961743 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1961743 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1961748 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1961748 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1961749 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1961749 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1961750 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@18f32b6b[dv_coll_shard3_replica_n4] main]
   [junit4]   2> 1961751 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1961752 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1961753 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@719c88fc[dv_coll_shard1_replica_n1] main]
   [junit4]   2> 1961753 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1961754 INFO  (searcherExecutor-8628-thread-1-processing-n:127.0.0.1:54174_solr x:dv_coll_shard4_replica_n6 s:shard4 c:dv_coll r:core_node8) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SolrCore [dv_coll_shard4_replica_n6] Registered new searcher Searcher@4f7e603f[dv_coll_shard4_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1961754 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1961754 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1961755 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1961756 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1961757 INFO  (searcherExecutor-8627-thread-1-processing-n:127.0.0.1:54175_solr x:dv_coll_shard3_replica_n4 s:shard3 c:dv_coll r:core_node7) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore [dv_coll_shard3_replica_n4] Registered new searcher Searcher@18f32b6b[dv_coll_shard3_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1961758 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148365542850560
   [junit4]   2> 1961764 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148365549142016
   [junit4]   2> 1961766 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1961770 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1961770 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1961771 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1961771 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1961772 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148365557530624
   [junit4]   2> 1961772 INFO  (searcherExecutor-8630-thread-1-processing-n:127.0.0.1:54175_solr x:dv_coll_shard1_replica_n1 s:shard1 c:dv_coll r:core_node3) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [dv_coll_shard1_replica_n1] Registered new searcher Searcher@719c88fc[dv_coll_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1961772 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1961772 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1961772 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54175/solr/dv_coll_shard3_replica_n4/
   [junit4]   2> 1961772 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1961772 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SyncStrategy http://127.0.0.1:54175/solr/dv_coll_shard3_replica_n4/ has no replicas
   [junit4]   2> 1961772 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1961778 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54175/solr/dv_coll_shard3_replica_n4/ shard3
   [junit4]   2> 1961782 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1961782 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1961782 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54175/solr/dv_coll_shard1_replica_n1/
   [junit4]   2> 1961782 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1961782 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:54175/solr/dv_coll_shard1_replica_n1/ has no replicas
   [junit4]   2> 1961782 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1961785 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1961785 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1961785 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54174/solr/dv_coll_shard4_replica_n6/
   [junit4]   2> 1961785 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1961785 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.SyncStrategy http://127.0.0.1:54174/solr/dv_coll_shard4_replica_n6/ has no replicas
   [junit4]   2> 1961785 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1961788 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54175/solr/dv_coll_shard1_replica_n1/ shard1
   [junit4]   2> 1961791 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54174/solr/dv_coll_shard4_replica_n6/ shard4
   [junit4]   2> 1961897 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1961897 INFO  (zkCallback-4813-thread-3-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1961897 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1961897 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1961933 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1961936 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf1&newCollection=true&name=dv_coll_shard3_replica_n4&action=CREATE&numShards=4&collection=dv_coll&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1833
   [junit4]   2> 1961949 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1961951 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1961953 INFO  (qtp897872790-19584) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf1&newCollection=true&name=dv_coll_shard2_replica_n2&action=CREATE&numShards=4&collection=dv_coll&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1852
   [junit4]   2> 1961953 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1961958 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=dv_coll_shard4_replica_n6&action=CREATE&numShards=4&collection=dv_coll&shard=shard4&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1854
   [junit4]   2> 1961961 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&newCollection=true&name=dv_coll_shard1_replica_n1&action=CREATE&numShards=4&collection=dv_coll&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1858
   [junit4]   2> 1961964 INFO  (qtp897872790-19580) [n:127.0.0.1:54174_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1962066 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1962066 INFO  (zkCallback-4813-thread-3-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1962066 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1962066 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/dv_coll/state.json] for collection [dv_coll] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1962972 INFO  (qtp897872790-19580) [n:127.0.0.1:54174_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&maxShardsPerNode=2&name=dv_coll&nrtReplicas=1&action=CREATE&numShards=4&wt=javabin&version=2} status=0 QTime=3222
   [junit4]   2> 1962980 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.SchemaManager managed schema loaded . version : 0
   [junit4]   2> 1962985 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.IndexSchema [dv_coll_shard3_replica_n4] Schema name=minimal
   [junit4]   2> 1962994 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963000 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.ZkController Persisted config data to node /configs/conf1/managed-schema
   [junit4]   2> 1963000 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 1963000 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 1963000 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 1963000 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/configs/conf1/managed-schema, has occurred - updating schema from ZooKeeper ...
   [junit4]   2> 1963001 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 1 from ZooKeeper
   [junit4]   2> 1963001 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 1 from ZooKeeper
   [junit4]   2> 1963001 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 1 from ZooKeeper
   [junit4]   2> 1963003 INFO  (Thread-4044) [n:127.0.0.1:54175_solr    ] o.a.s.c.SolrCore config update listener called for core dv_coll_shard1_replica_n1
   [junit4]   2> 1963003 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.IndexSchema [dv_coll_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 1963004 INFO  (Thread-4045) [n:127.0.0.1:54174_solr    ] o.a.s.c.SolrCore config update listener called for core dv_coll_shard2_replica_n2
   [junit4]   2> 1963003 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.IndexSchema [dv_coll_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1963004 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.IndexSchema [dv_coll_shard4_replica_n6] Schema name=minimal
   [junit4]   2> 1963009 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963009 INFO  (zkCallback-4812-thread-1-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.ZkIndexSchemaReader Finished refreshing schema in 8 ms
   [junit4]   2> 1963009 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963009 INFO  (zkCallback-4813-thread-2-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.ZkIndexSchemaReader Finished refreshing schema in 8 ms
   [junit4]   2> 1963010 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963010 INFO  (zkCallback-4812-thread-2-processing-n:127.0.0.1:54174_solr) [n:127.0.0.1:54174_solr    ] o.a.s.s.ZkIndexSchemaReader Finished refreshing schema in 8 ms
   [junit4]   2> 1963012 INFO  (Thread-4045) [n:127.0.0.1:54174_solr    ] o.a.s.c.SolrCore core reload dv_coll_shard2_replica_n2
   [junit4]   2> 1963012 INFO  (Thread-4044) [n:127.0.0.1:54175_solr    ] o.a.s.c.SolrCore core reload dv_coll_shard1_replica_n1
   [junit4]   2> 1963022 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1963022 INFO  (Thread-4044) [n:127.0.0.1:54175_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1963022 INFO  (Thread-4045) [n:127.0.0.1:54174_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1963056 INFO  (Thread-4045) [n:127.0.0.1:54174_solr    ] o.a.s.s.IndexSchema [dv_coll_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 1963056 INFO  (Thread-4044) [n:127.0.0.1:54175_solr    ] o.a.s.s.IndexSchema [dv_coll_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1963059 INFO  (Thread-4044) [n:127.0.0.1:54175_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963059 INFO  (Thread-4045) [n:127.0.0.1:54174_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963059 INFO  (Thread-4044) [n:127.0.0.1:54175_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'dv_coll_shard1_replica_n1' using configuration from collection dv_coll
   [junit4]   2> 1963059 INFO  (Thread-4045) [n:127.0.0.1:54174_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'dv_coll_shard2_replica_n2' using configuration from collection dv_coll
   [junit4]   2> 1963065 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.IndexSchema [dv_coll_shard3_replica_n4] Schema name=minimal
   [junit4]   2> 1963067 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963067 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.CoreContainer Reloading SolrCore 'dv_coll_shard3_replica_n4' using configuration from collection dv_coll
   [junit4]   2> 1963167 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54174.solr.core.dv_coll.shard2.replica_n2' (registry 'solr.core.dv_coll.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1963167 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore [[dv_coll_shard2_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/dv_coll_shard2_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node1/./dv_coll_shard2_replica_n2/data/]
   [junit4]   2> 1963171 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.core.dv_coll.shard1.replica_n1' (registry 'solr.core.dv_coll.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1963171 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [[dv_coll_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/dv_coll_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/./dv_coll_shard1_replica_n1/data/]
   [junit4]   2> 1963185 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.core.dv_coll.shard3.replica_n4' (registry 'solr.core.dv_coll.shard3.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1963185 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore [[dv_coll_shard3_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/dv_coll_shard3_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/./dv_coll_shard3_replica_n4/data/]
   [junit4]   2> 1963232 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1963232 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1963233 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ce0fce5[dv_coll_shard2_replica_n2] main]
   [junit4]   2> 1963235 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1963235 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1963235 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1963235 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1963235 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1963236 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@46dbfa19[dv_coll_shard1_replica_n1] main]
   [junit4]   2> 1963236 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 1 is already the latest
   [junit4]   2> 1963237 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1963237 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1963238 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1963238 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1963238 INFO  (searcherExecutor-8648-thread-1-processing-n:127.0.0.1:54174_solr x:dv_coll_shard2_replica_n2 s:shard2 c:dv_coll r:core_node5) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore [dv_coll_shard2_replica_n2] Registered new searcher Searcher@5ce0fce5[dv_coll_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1963239 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 1 is already the latest
   [junit4]   2> 1963239 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148367095791616
   [junit4]   2> 1963239 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1963240 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148367096840192
   [junit4]   2> 1963243 INFO  (searcherExecutor-8647-thread-1-processing-n:127.0.0.1:54175_solr x:dv_coll_shard1_replica_n1 s:shard1 c:dv_coll r:core_node3) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [dv_coll_shard1_replica_n1] Registered new searcher Searcher@46dbfa19[dv_coll_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1963247 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1963247 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1963247 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@17a271b5[dv_coll_shard2_replica_n2] main]
   [junit4]   2> 1963247 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore [dv_coll_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@34f3ea9f
   [junit4]   2> 1963247 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.dv_coll.shard2.replica_n2, tag=888400543
   [junit4]   2> 1963247 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ab93e6: rootName = solr_54174, domain = solr.core.dv_coll.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.dv_coll.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@5ec6b1bd
   [junit4]   2> 1963248 INFO  (searcherExecutor-8648-thread-1-processing-n:127.0.0.1:54174_solr x:dv_coll_shard2_replica_n2 s:shard2 c:dv_coll r:core_node5) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore [dv_coll_shard2_replica_n2] Registered new searcher Searcher@17a271b5[dv_coll_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1963251 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.dv_coll.shard2.leader, tag=888400543
   [junit4]   2> 1963253 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1963286 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1963251 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4aa9bfef[dv_coll_shard1_replica_n1] main]
   [junit4]   2> 1963287 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [dv_coll_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@25cc641d
   [junit4]   2> 1963287 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.dv_coll.shard1.replica_n1, tag=634151965
   [junit4]   2> 1963287 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5023dd3d: rootName = solr_54175, domain = solr.core.dv_coll.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.dv_coll.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@46712fa6
   [junit4]   2> 1963299 INFO  (searcherExecutor-8647-thread-1-processing-n:127.0.0.1:54175_solr x:dv_coll_shard1_replica_n1 s:shard1 c:dv_coll r:core_node3) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [dv_coll_shard1_replica_n1] Registered new searcher Searcher@4aa9bfef[dv_coll_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1963313 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@681b3ee5[dv_coll_shard3_replica_n4] main]
   [junit4]   2> 1963315 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1963317 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1963317 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 1963317 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.dv_coll.shard1.leader, tag=634151965
   [junit4]   2> 1963318 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.ZkIndexSchemaReader Current schema version 1 is already the latest
   [junit4]   2> 1963319 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1963321 INFO  (searcherExecutor-8649-thread-1-processing-n:127.0.0.1:54175_solr x:dv_coll_shard3_replica_n4 s:shard3 c:dv_coll r:core_node7) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore [dv_coll_shard3_replica_n4] Registered new searcher Searcher@681b3ee5[dv_coll_shard3_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1963324 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591148367184920576
   [junit4]   2> 1963333 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore config update listener called for core dv_coll_shard3_replica_n4
   [junit4]   2> 1963335 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore core reload dv_coll_shard3_replica_n4
   [junit4]   2> 1963335 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1963336 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@3603340c[dv_coll_shard3_replica_n4] main]
   [junit4]   2> 1963338 INFO  (searcherExecutor-8649-thread-1-processing-n:127.0.0.1:54175_solr x:dv_coll_shard3_replica_n4 s:shard3 c:dv_coll r:core_node7) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore [dv_coll_shard3_replica_n4] Registered new searcher Searcher@3603340c[dv_coll_shard3_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1963338 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 1 from ZooKeeper
   [junit4]   2> 1963343 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.IndexSchema [dv_coll_shard3_replica_n4] Schema name=minimal
   [junit4]   2> 1963347 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1963348 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963348 INFO  (zkCallback-4813-thread-1-processing-n:127.0.0.1:54175_solr) [n:127.0.0.1:54175_solr    ] o.a.s.s.ZkIndexSchemaReader Finished refreshing schema in 9 ms
   [junit4]   2> 1963351 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.ManagedIndexSchema Waiting up to 599 secs for 3 replicas to apply schema update version 1 for collection dv_coll
   [junit4]   2> 1963411 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.IndexSchema [dv_coll_shard3_replica_n4] Schema name=minimal
   [junit4]   2> 1963413 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard4 r:core_node8 x:dv_coll_shard4_replica_n6] o.a.s.c.S.Request [dv_coll_shard4_replica_n6]  webapp=/solr path=/schema/zkversion params={refreshIfBelowVersion=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1963414 INFO  (qtp836209285-19591) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.S.Request [dv_coll_shard1_replica_n1]  webapp=/solr path=/schema/zkversion params={refreshIfBelowVersion=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1963415 INFO  (qtp897872790-19586) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.S.Request [dv_coll_shard2_replica_n2]  webapp=/solr path=/schema/zkversion params={refreshIfBelowVersion=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1963416 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.s.ManagedIndexSchema Took 77.0ms for 3 replicas to apply schema update version 1 for collection dv_coll
   [junit4]   2> 1963416 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1963416 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.S.Request [dv_coll_shard3_replica_n4]  webapp=/solr path=/schema params={wt=javabin&version=2} status=0 QTime=436
   [junit4]   2> 1963417 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore [dv_coll_shard3_replica_n4]  CLOSING SolrCore org.apache.solr.core.SolrCore@9f9f116
   [junit4]   2> 1963417 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.dv_coll.shard3.replica_n4, tag=167375126
   [junit4]   2> 1963417 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2fd6d7c6: rootName = solr_54175, domain = solr.core.dv_coll.shard3.replica_n4, service url = null, agent id = null] for registry solr.core.dv_coll.shard3.replica_n4 / com.codahale.metrics.MetricRegistry@1510e3b3
   [junit4]   2> 1963416 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.CoreContainer Reloading SolrCore 'dv_coll_shard3_replica_n4' using configuration from collection dv_coll
   [junit4]   2> 1963419 INFO  (Thread-4045) [n:127.0.0.1:54174_solr c:dv_coll s:shard2 r:core_node5 x:dv_coll_shard2_replica_n2] o.a.s.c.SolrCore config update listener called for core dv_coll_shard4_replica_n6
   [junit4]   2> 1963425 INFO  (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.dv_coll.shard3.leader, tag=167375126
   [junit4]   2> 1963453 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54175.solr.core.dv_coll.shard3.replica_n4' (registry 'solr.core.dv_coll.shard3.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e62e693
   [junit4]   2> 1963453 INFO  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.c.SolrCore [[dv_coll_shard3_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/dv_coll_shard3_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DocValuesNotIndexedTest_2EB9A973366553B3-001/tempDir-001/node2/./dv_coll_shard3_replica_n4/data/]
   [junit4]   2> 1963495 WARN  (Thread-4044) [n:127.0.0.1:54175_solr c:dv_coll s:shard1 r:core_node3 x:dv_coll_shard1_replica_n1] o.a.s.m.r.j.JmxMetricsReporter Unable to register gauge
   [junit4]   2> javax.management.InstanceNotFoundException: solr_54175:dom1=solr,dom2=core,dom3=dv_coll,dom4=shard3,dom5=replica_n4,category=UPDATE,scope=update,name=handlerStart
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.solr.metrics.reporters.jmx.JmxMetricsReporter$JmxListener.registerMBean(JmxMetricsReporter.java:530)
   [junit4]   2> at org.apache.solr.metrics.reporters.jmx.JmxMetricsReporter$JmxListener.onGaugeAdded(JmxMetricsReporter.java:565)
   [junit4]   2> at com.codahale.metrics.MetricRegistry.notifyListenerOfAddedMetric(MetricRegistry.java:454)
   [junit4]   2> at com.codahale.metrics.MetricRegistry.onMetricAdded(MetricRegistry.java:448)
   [junit4]   2> at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:89)
   [junit4]   2> at org.apache.solr.metrics.SolrMetricManager.register(SolrMetricManager.java:658)
   [junit4]   2> at org.apache.solr.metrics.SolrMetricManager.registerGauge(SolrMetricManager.java:663)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.initializeMetrics(RequestHandlerBase.java:151)
   [junit4]   2> at org.apache.solr.metrics.SolrCoreMetricManager.registerMetricProducer(SolrCoreMetricManager.java:131)
   [junit4]   2> at org.apache.solr.core.SolrCore.registerInfoBean(SolrCore.java:3001)
   [junit4]   2> at org.apache.solr.core.PluginBag.registerMBean(PluginBag.java:302)
   [junit4]   2> at org.apache.solr.core.PluginBag.put(PluginBag.java:232)
   [junit4]   2> at org.apache.solr.core.PluginBag.init(PluginBag.java:274)
   [junit4]   2> at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
   [junit4]   2> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:966)
   [junit4]   2> at org.apache.solr.core.SolrCore.reload(SolrCore.java:656)
   [junit4]   2> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1291)
   [junit4]   2> at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:2969)
   [junit4]   2> at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$5(ZkController.java:2610)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1963495 ERROR (qtp836209285-19589) [n:127.0.0.1:54175_solr c:dv_coll s:shard3 r:core_node7 x:dv_coll_shard3_replica_n4] o.a.s.c.SolrCore java.lang.IllegalArgumentException: A metric named UPDATE.update.handlerStart already exists
   [junit4]   2> at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91)
   [junit4]   2> at org.apache.solr.metrics.SolrCoreMetricManager.lambda$close$1(SolrCoreMetricManager.java:158)
   [junit4]   2> at java.util.TreeMap.forEach(TreeMap.java:1005)
   [junit4]   2> at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1505)
   [junit4]   2> at org.apache.solr.metrics.SolrCoreMetricManager.close(SolrCoreMetricManager.java:155)
   [junit4]   2> at org.apache.solr.core.SolrCore.close(SolrCore.java:1513)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:570)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFi

[...truncated too long message...]

, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3090621 INFO  (SUITE-TriggerIntegrationTest-seed#[2EB9A973366553B3]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58132 58132
   [junit4]   2> 3090652 INFO  (Thread-5158) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58132 58132
   [junit4]   2> 3090863 WARN  (Thread-5158) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 22 /solr/clusterprops.json
   [junit4]   2> 22 /solr/aliases.json
   [junit4]   2> 21 /solr/security.json
   [junit4]   2> 21 /solr/collections/collection1/terms/shard1
   [junit4]   2> 6 /solr/configs/conf
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard1
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard2
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 71 /solr/collections/collection1/state.json
   [junit4]   2> 22 /solr/clusterstate.json
   [junit4]   2> 11 /solr/collections/testMetricTrigger/state.json
   [junit4]   2> 6 /solr/autoscaling.json
   [junit4]   2> 3 /solr/overseer_elect/election/72119989887107116-127.0.0.1:58289_solr-n_0000000013
   [junit4]   2> 3 /solr/collections/collection1/leader_elect/shard1/election/72119989887107110-core_node3-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72119989887107089-127.0.0.1:58160_solr-n_0000000004
   [junit4]   2> 2 /solr/overseer_elect/election/72119989887107077-127.0.0.1:58134_solr-n_0000000001
   [junit4]   2> 2 /solr/overseer_elect/election/72119989887107104-127.0.0.1:58231_solr-n_0000000009
   [junit4]   2> 2 /solr/overseer_elect/election/72119989887107110-127.0.0.1:58245_solr-n_0000000011
   [junit4]   2> 2 /solr/overseer_elect/election/72119989887107128-127.0.0.1:58401_solr-n_0000000017
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 22 /solr/live_nodes
   [junit4]   2> 22 /solr/collections
   [junit4]   2> 6 /solr/overseer/queue-work
   [junit4]   2> 6 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 6 /solr/overseer/queue
   [junit4]   2> 6 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/autoscaling/events/node_lost_triggerMR
   [junit4]   2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_2EB9A973366553B3-001
   [junit4]   2> Jan 31, 2018 10:40:58 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=MockRandom), x_s=PostingsFormat(name=Memory)}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=614, maxMBSortInHeap=5.018918098613172, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@284e5dc8), locale=de-DE, timezone=America/Coral_Harbour
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=93061224,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestMinMaxOnMultiValuedField, DistribDocExpirationUpdateProcessorTest, TestRuleBasedAuthorizationPlugin, TestLuceneIndexBackCompat, DeleteStatusTest, SolrCoreMetricManagerTest, SolrShardReporterTest, BasicFunctionalityTest, BaseCdcrDistributedZkTest, DocValuesMultiTest, BasicAuthStandaloneTest, TestCloudManagedSchema, TestSimDistributedQueue, DocValuesTest, SolrCmdDistributorTest, HdfsBasicDistributedZk2Test, TestRawTransformer, TestHashPartitioner, TestNodeLostTrigger, PreAnalyzedUpdateProcessorTest, DistributedSpellCheckComponentTest, SchemaApiFailureTest, ChaosMonkeySafeLeaderTest, TestSolrCLIRunExample, ReplicationFactorTest, SimpleCollectionCreateDeleteTest, TestRandomRequestDistribution, NodeLostTriggerTest, TestRandomDVFaceting, TestSort, TestNRTOpen, TestFieldCacheSortRandom, TestComputePlanAction, HealthCheckHandlerTest, CustomHighlightComponentTest, SliceStateTest, TestLeaderInitiatedRecoveryThread, DataDrivenBlockJoinTest, SyncSliceTest, BadIndexSchemaTest, SoftAutoCommitTest, TestInfoStreamLogging, TestDynamicFieldCollectionResource, AliasIntegrationTest, SuggesterTest, TestSolrCloudSnapshots, PrimitiveFieldTypeTest, OutOfBoxZkACLAndCredentialsProvidersTest, AutoAddReplicasPlanActionTest, TestPullReplicaErrorHandling, ConnectionManagerTest, TestUnifiedSolrHighlighter, TestSolrCloudWithDelegationTokens, FieldAnalysisRequestHandlerTest, RuleEngineTest, TlogReplayBufferedWhileIndexingTest, XsltUpdateRequestHandlerTest, TestJsonRequest, ClassificationUpdateProcessorIntegrationTest, NumberUtilsTest, JsonLoaderTest, TimeRoutedAliasUpdateProcessorTest, SimpleMLTQParserTest, TestQuerySenderListener, TestSolrQueryParser, TestExecutePlanAction, TestCustomSort, TestLMDirichletSimilarityFactory, TestCloudJSONFacetJoinDomain, TestComplexPhraseLeadingWildcard, TestCollationFieldDocValues, DistributedQueueTest, LeaderElectionContextKeyTest, TestSerializedLuceneMatchVersion, PreAnalyzedFieldTest, TestPushWriter, TestCodecSupport, TestExceedMaxTermLength, TestSchemaNameResource, TestFieldCollectionResource, SimplePostToolTest, ReplaceNodeNoTargetTest, ConjunctionSolrSpellCheckerTest, TestUpdate, ClusterStateTest, JavabinLoaderTest, DistributedTermsComponentTest, TestSortByMinMaxFunction, TestJettySolrRunner, ChaosMonkeySafeLeaderWithPullReplicasTest, TestLRUCache, TestDistributedMissingSort, TestDeleteCollectionOnDownNodes, RequestHandlersTest, ReturnFieldsTest, SolrGangliaReporterTest, RecoveryAfterSoftCommitTest, TestTolerantUpdateProcessorRandomCloud, MissingSegmentRecoveryTest, TestLegacyTerms, TestDocSet, FacetPivotSmallTest, CSVRequestHandlerTest, QueryResultKeyTest, SolrCoreCheckLockOnStartupTest, BitVectorTest, TestLegacyNumericRangeQueryBuilder, UninvertDocValuesMergePolicyTest, TestManagedSchema, DefaultValueUpdateProcessorTest, TestComponentsName, TestDistributedGrouping, OverseerCollectionConfigSetProcessorTest, SolrMetricReporterTest, TestLegacyFieldReuse, CoreSorterTest, TestNoOpRegenerator, TestRecoveryHdfs, TestNumericTerms32, SolrCLIZkUtilsTest, TestStressVersions, SuggestComponentTest, TestWordDelimiterFilterFactory, DistributedFacetPivotSmallAdvancedTest, HdfsTlogReplayBufferedWhileIndexingTest, TestConfigsApi, SolrJmxReporterCloudTest, TestLeaderElectionWithEmptyReplica, TestSolrIndexConfig, TestNumericTokenStream, HLLUtilTest, TestFieldCacheWithThreads, AnalyticsMergeStrategyTest, TestSubQueryTransformer, UpdateLogTest, TestPrepRecovery, CdcrBootstrapTest, BigEndianAscendingWordDeserializerTest, BlobRepositoryCloudTest, TestCollectionsAPIViaSolrCloudCluster, TestSubQueryTransformerDistrib, TestCoreDiscovery, JvmMetricsTest, TestSimpleTrackingShardHandler, TestStressInPlaceUpdates, TestAnalyzeInfixSuggestions, TestPerFieldSimilarityWithDefaultOverride, CoreAdminHandlerTest, BasicDistributedZk2Test, MoveReplicaHDFSTest, LargeFieldTest, InfoHandlerTest, ManagedSchemaRoundRobinCloudTest, SolrXmlInZkTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, JSONWriterTest, TestUseDocValuesAsStored, SubstringBytesRefFilterTest, ReplicaListTransformerTest, TestCloudNestedDocsSort, SuggesterWFSTTest, SaslZkACLProviderTest, PeerSyncTest, TestSolrConfigHandler, OverseerTest, BlockDirectoryTest, ZkControllerTest, TermVectorComponentTest, TestCloudDeleteByQuery, URLClassifyProcessorTest, TestGraphMLResponseWriter, TestDFISimilarityFactory, TestFilteredDocIdSet, TestFileDictionaryLookup, SpatialHeatmapFacetsTest, DisMaxRequestHandlerTest, TestIndexingPerformance, CurrencyRangeFacetCloudTest, LoggingHandlerTest, SpellPossibilityIteratorTest, TestPHPSerializedResponseWriter, TestManagedResourceStorage, RegexBoostProcessorTest, DocValuesMissingTest, TestConfigSetsAPI, AutoscalingHistoryHandlerTest, TestSolrCoreParser, TestManagedSynonymGraphFilterFactory, TestScoreJoinQPNoScore, TestSolrCloudWithHadoopAuthPlugin, TestFunctionQuery, DirectUpdateHandlerTest, TestBadConfig, HighlighterTest, ShowFileRequestHandlerTest, TestExtendedDismaxParser, SpellCheckCollatorTest, TestStressLucene, TestCSVLoader, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, StatelessScriptUpdateProcessorFactoryTest, DistanceFunctionTest, DebugComponentTest, IndexBasedSpellCheckerTest, TestQueryUtils, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, TermsComponentTest, QueryParsingTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, SearchHandlerTest, TestLFUCache, HighlighterConfigTest, UpdateParamsTest, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, ChaosMonkeyNothingIsSafeTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, ConcurrentCreateRoutedAliasTest, CreateCollectionCleanupTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteShardTest, DistribCursorPagingTest, DocValuesNotIndexedTest, ForceLeaderTest, LeaderFailureAfterFreshStartTest, NodeMutatorTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerRolesTest, OverseerTaskQueueTest, PeerSyncReplicationTest, TestAuthenticationFramework, TestCloudPivotFacet, TestConfigSetsAPIZkFailure, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry, TestLockTree, TestMiniSolrCloudClusterSSL, TestPullReplica, TestTlogReplica, ConcurrentDeleteAndCreateCollectionTest, HdfsCollectionsAPIDistributedZkTest, ShardSplitTest, ComputePlanActionTest, SystemLogListenerTest, TriggerIntegrationTest]
   [junit4] Completed [621/774 (2!)] on J1 in 192.96s, 14 tests, 1 failure <<< FAILURES!

[...truncated 48640 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4418 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4418/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 8 in http://127.0.0.1:50137/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 8 in http://127.0.0.1:50137/solr
        at __randomizedtesting.SeedInfo.seed([624230E011B6037C:A3B2494C3CE6C9DB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:885)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:599)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost

Error Message:
The operations computed by ComputePlanAction should not be null SolrClientNodeStateProvider.DEBUG{AFTER_ACTION=[compute_plan, null], BEFORE_ACTION=[compute_plan, null]}

Stack Trace:
java.lang.AssertionError: The operations computed by ComputePlanAction should not be null SolrClientNodeStateProvider.DEBUG{AFTER_ACTION=[compute_plan, null], BEFORE_ACTION=[compute_plan, null]}
        at __randomizedtesting.SeedInfo.seed([624230E011B6037C:5282D16299C4E220]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost(ComputePlanActionTest.java:291)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([624230E011B6037C:EA7FB99F2B76E2D1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState(TriggerIntegrationTest.java:426)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTrigger

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([624230E011B6037C:66BC99F22B7C81E6]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTrigger(TriggerIntegrationTest.java:521)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 13465 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/init-core-data-001
   [junit4]   2> 2720679 WARN  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2720679 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2720680 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2720680 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2720680 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001
   [junit4]   2> 2720680 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2720681 INFO  (Thread-4339) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2720681 INFO  (Thread-4339) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2720683 ERROR (Thread-4339) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2720786 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:50135
   [junit4]   2> 2720798 INFO  (zkConnectionManagerCallback-10156-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720803 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2720803 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2720804 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2720804 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2720804 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2720805 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@332bcb7a{/solr,null,AVAILABLE}
   [junit4]   2> 2720805 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2720805 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2720805 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2720805 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4667b60{HTTP/1.1,[http/1.1]}{127.0.0.1:50137}
   [junit4]   2> 2720805 INFO  (jetty-launcher-10153-thread-2) [    ] o.e.j.s.Server Started @2722932ms
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50137}
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7cbf5e07{/solr,null,AVAILABLE}
   [junit4]   2> 2720806 ERROR (jetty-launcher-10153-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1e763009{HTTP/1.1,[http/1.1]}{127.0.0.1:50138}
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-1) [    ] o.e.j.s.Server Started @2722933ms
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50138}
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2720806 ERROR (jetty-launcher-10153-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-01T08:59:46.537709Z
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2720806 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2720807 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2720807 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-01T08:59:46.538060Z
   [junit4]   2> 2720808 INFO  (zkConnectionManagerCallback-10158-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720808 INFO  (zkConnectionManagerCallback-10160-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720809 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2720809 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2720826 INFO  (jetty-launcher-10153-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50135/solr
   [junit4]   2> 2720826 INFO  (jetty-launcher-10153-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50135/solr
   [junit4]   2> 2720827 INFO  (zkConnectionManagerCallback-10165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720828 INFO  (zkConnectionManagerCallback-10168-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720830 INFO  (zkConnectionManagerCallback-10170-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720831 INFO  (zkConnectionManagerCallback-10172-thread-1-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2720976 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2720977 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50137_solr
   [junit4]   2> 2720978 INFO  (jetty-launcher-10153-thread-1) [n:127.0.0.1:50138_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2720978 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.c.Overseer Overseer (id=72122435697967110-127.0.0.1:50137_solr-n_0000000000) starting
   [junit4]   2> 2720979 INFO  (jetty-launcher-10153-thread-1) [n:127.0.0.1:50138_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50138_solr
   [junit4]   2> 2720981 INFO  (zkCallback-10171-thread-1-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2720981 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2720993 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50137_solr
   [junit4]   2> 2720995 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2720995 INFO  (zkCallback-10171-thread-1-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2721046 INFO  (jetty-launcher-10153-thread-1) [n:127.0.0.1:50138_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50138.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2721057 INFO  (jetty-launcher-10153-thread-1) [n:127.0.0.1:50138_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50138.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2721057 INFO  (jetty-launcher-10153-thread-1) [n:127.0.0.1:50138_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50138.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2721057 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50137.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2721058 INFO  (jetty-launcher-10153-thread-1) [n:127.0.0.1:50138_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node1/.
   [junit4]   2> 2721066 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50137.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2721066 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50137.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2721067 INFO  (jetty-launcher-10153-thread-2) [n:127.0.0.1:50137_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node2/.
   [junit4]   2> 2721099 INFO  (zkConnectionManagerCallback-10178-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2721103 INFO  (zkConnectionManagerCallback-10182-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2721105 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2721107 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50135/solr ready
   [junit4]   2> 2721114 INFO  (SUITE-TestTlogReplica-seed#[624230E011B6037C]-worker) [    ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
   [junit4]   2> 2721117 INFO  (qtp1387007808-31122) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2721118 INFO  (qtp1387007808-31122) [n:127.0.0.1:50137_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2721129 INFO  (TEST-TestTlogReplica.testAddDocs-seed#[624230E011B6037C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAddDocs
   [junit4]   2> 2721131 INFO  (qtp1387007808-31124) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_add_docs&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2721135 INFO  (OverseerThreadFactory-10289-thread-1-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection tlog_replica_test_add_docs
   [junit4]   2> 2721254 INFO  (OverseerStateUpdate-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_add_docs",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_add_docs_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:50138/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2721470 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=tlog_replica_test_add_docs_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_add_docs&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2721470 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2721576 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_docs/state.json] for collection [tlog_replica_test_add_docs] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2721577 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_docs/state.json] for collection [tlog_replica_test_add_docs] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2722487 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2722492 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_add_docs_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 2722493 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2722493 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_add_docs_shard1_replica_t1' using configuration from collection tlog_replica_test_add_docs, trusted=true
   [junit4]   2> 2722493 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50138.solr.core.tlog_replica_test_add_docs.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_add_docs.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2722493 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2722493 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_add_docs_shard1_replica_t1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node1/tlog_replica_test_add_docs_shard1_replica_t1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node1/./tlog_replica_test_add_docs_shard1_replica_t1/data/]
   [junit4]   2> 2722537 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2722537 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2722538 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2722538 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2722539 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ed985a3[tlog_replica_test_add_docs_shard1_replica_t1] main]
   [junit4]   2> 2722541 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2722541 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2722542 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2722542 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591188482448949248
   [junit4]   2> 2722543 INFO  (searcherExecutor-10294-thread-1-processing-n:127.0.0.1:50138_solr x:tlog_replica_test_add_docs_shard1_replica_t1 s:shard1 c:tlog_replica_test_add_docs r:core_node2) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_add_docs_shard1_replica_t1] Registered new searcher Searcher@5ed985a3[tlog_replica_test_add_docs_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50138/solr/tlog_replica_test_add_docs_shard1_replica_t1/
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SyncStrategy http://127.0.0.1:50138/solr/tlog_replica_test_add_docs_shard1_replica_t1/ has no replicas
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2722553 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_add_docs_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 2722558 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:50138/solr/tlog_replica_test_add_docs_shard1_replica_t1/ shard1
   [junit4]   2> 2722669 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_docs/state.json] for collection [tlog_replica_test_add_docs] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2722669 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_docs/state.json] for collection [tlog_replica_test_add_docs] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2722672 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2722675 INFO  (qtp469824676-31126) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=tlog_replica_test_add_docs_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_add_docs&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1205
   [junit4]   2> 2722679 INFO  (qtp1387007808-31124) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2722784 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_docs/state.json] for collection [tlog_replica_test_add_docs] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2722784 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_add_docs/state.json] for collection [tlog_replica_test_add_docs] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2723147 INFO  (OverseerCollectionConfigSetProcessor-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2723690 INFO  (qtp1387007808-31124) [n:127.0.0.1:50137_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_add_docs&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2} status=0 QTime=2559
   [junit4]   2> 2723696 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_add_docs_shard1_replica_t1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1591188483657957376)]} 0 1
   [junit4]   2> 2723697 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1591188483660054528,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2723697 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@38c70f4d commitCommandVersion:1591188483660054528
   [junit4]   2> 2723700 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6ee5caa7[tlog_replica_test_add_docs_shard1_replica_t1] main]
   [junit4]   2> 2723701 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2723702 INFO  (searcherExecutor-10294-thread-1-processing-n:127.0.0.1:50138_solr x:tlog_replica_test_add_docs_shard1_replica_t1 s:shard1 c:tlog_replica_test_add_docs r:core_node2) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_add_docs_shard1_replica_t1] Registered new searcher Searcher@6ee5caa7[tlog_replica_test_add_docs_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 2723702 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_add_docs_shard1_replica_t1]  webapp=/solr path=/update params={_stateVer_=tlog_replica_test_add_docs:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 5
   [junit4]   2> 2723704 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_add_docs_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 2723706 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_add_docs_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 2723711 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_add_docs s:shard1 r:core_node2 x:tlog_replica_test_add_docs_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_add_docs_shard1_replica_t1]  webapp=/solr path=/admin/plugins params={qt=/admin/plugins&stats=true&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 2723712 INFO  (TEST-TestTlogReplica.testAddDocs-seed#[624230E011B6037C]) [    ] o.a.s.c.TestTlogReplica tearDown deleting collection
   [junit4]   2> 2723713 INFO  (qtp1387007808-31127) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=tlog_replica_test_add_docs&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2723717 INFO  (OverseerThreadFactory-10289-thread-2-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 2723718 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_add_docs.shard1.replica_t1, tag=null
   [junit4]   2> 2723718 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4eaa0f50: rootName = solr_50138, domain = solr.core.tlog_replica_test_add_docs.shard1.replica_t1, service url = null, agent id = null] for registry solr.core.tlog_replica_test_add_docs.shard1.replica_t1 / com.codahale.metrics.MetricRegistry@eb9230d
   [junit4]   2> 2723728 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.c.ZkController tlog_replica_test_add_docs_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 2723728 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.c.SolrCore [tlog_replica_test_add_docs_shard1_replica_t1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4fae1bb2
   [junit4]   2> 2723728 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_add_docs.shard1.replica_t1, tag=1336810418
   [junit4]   2> 2723729 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_add_docs.shard1.leader, tag=1336810418
   [junit4]   2> 2723733 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_add_docs_shard1_replica_t1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=15
   [junit4]   2> 2724454 INFO  (qtp1387007808-31127) [n:127.0.0.1:50137_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=tlog_replica_test_add_docs&action=DELETE&wt=javabin&version=2} status=0 QTime=740
   [junit4]   2> 2724454 INFO  (TEST-TestTlogReplica.testAddDocs-seed#[624230E011B6037C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testAddDocs
   [junit4]   2> 2724472 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRecovery
   [junit4]   2> 2724473 INFO  (qtp1387007808-31124) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_recovery&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2724476 INFO  (OverseerThreadFactory-10289-thread-3-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection tlog_replica_test_recovery
   [junit4]   2> 2724477 INFO  (OverseerCollectionConfigSetProcessor-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2724691 INFO  (OverseerStateUpdate-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:50137/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2724695 INFO  (OverseerStateUpdate-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_recovery",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_recovery_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:50138/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2724909 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2724909 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2724909 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2725040 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2725040 INFO  (zkCallback-10171-thread-1-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2725040 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2725041 INFO  (zkCallback-10171-thread-2-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2725954 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2725955 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2725961 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_recovery_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 2725963 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2725963 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_recovery_shard1_replica_t2' using configuration from collection tlog_replica_test_recovery, trusted=true
   [junit4]   2> 2725963 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50138.solr.core.tlog_replica_test_recovery.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_recovery.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2725963 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2725963 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_recovery_shard1_replica_t2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node1/tlog_replica_test_recovery_shard1_replica_t2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node1/./tlog_replica_test_recovery_shard1_replica_t2/data/]
   [junit4]   2> 2725964 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_recovery_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 2725965 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2725965 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_recovery_shard1_replica_t1' using configuration from collection tlog_replica_test_recovery, trusted=true
   [junit4]   2> 2725966 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50137.solr.core.tlog_replica_test_recovery.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_recovery.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d8cbbcc
   [junit4]   2> 2725966 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2725966 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_recovery_shard1_replica_t1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node2/tlog_replica_test_recovery_shard1_replica_t1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_624230E011B6037C-001/tempDir-001/node2/./tlog_replica_test_recovery_shard1_replica_t1/data/]
   [junit4]   2> 2726055 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2726055 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2726056 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2726056 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2726059 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2726059 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2726063 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2726063 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2726089 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@62beb4a[tlog_replica_test_recovery_shard1_replica_t2] main]
   [junit4]   2> 2726090 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2726091 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2726091 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2726092 INFO  (searcherExecutor-10299-thread-1-processing-n:127.0.0.1:50138_solr x:tlog_replica_test_recovery_shard1_replica_t2 s:shard1 c:tlog_replica_test_recovery r:core_node4) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t2] Registered new searcher Searcher@62beb4a[tlog_replica_test_recovery_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2726092 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591188486171394048
   [junit4]   2> 2726104 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2726105 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@596d83f8[tlog_replica_test_recovery_shard1_replica_t1] main]
   [junit4]   2> 2726107 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2726109 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2726110 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2726111 INFO  (searcherExecutor-10300-thread-1-processing-n:127.0.0.1:50137_solr x:tlog_replica_test_recovery_shard1_replica_t1 s:shard1 c:tlog_replica_test_recovery r:core_node3) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t1] Registered new searcher Searcher@596d83f8[tlog_replica_test_recovery_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2726111 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591188486191316992
   [junit4]   2> 2726555 INFO  (zkCallback-10171-thread-3-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2726555 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2726555 INFO  (zkCallback-10171-thread-2-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2726555 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2726608 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2726608 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2726608 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50138/solr/tlog_replica_test_recovery_shard1_replica_t2/
   [junit4]   2> 2726609 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_recovery_shard1_replica_t2 url=http://127.0.0.1:50138/solr START replicas=[http://127.0.0.1:50137/solr/tlog_replica_test_recovery_shard1_replica_t1/] nUpdates=100
   [junit4]   2> 2726611 INFO  (qtp1387007808-31130) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2726911 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_recovery_shard1_replica_t2 url=http://127.0.0.1:50138/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2726911 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2726911 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2726911 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2726912 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_recovery_shard1_replica_t2 stopping background replication from leader
   [junit4]   2> 2726917 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:50138/solr/tlog_replica_test_recovery_shard1_replica_t2/ shard1
   [junit4]   2> 2727021 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727021 INFO  (zkCallback-10171-thread-2-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727021 INFO  (zkCallback-10171-thread-3-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727021 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727070 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2727071 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2162
   [junit4]   2> 2727129 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_recovery_shard1_replica_t1 starting background replication from leader
   [junit4]   2> 2727129 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 2727130 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 2727130 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2727130 INFO  (indexFetcher-10310-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.h.IndexFetcher Replica core_node4 is leader but it's state is down, skipping replication
   [junit4]   2> 2727132 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_recovery_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_recovery&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2223
   [junit4]   2> 2727135 INFO  (qtp1387007808-31124) [n:127.0.0.1:50137_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2727239 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727239 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727240 INFO  (zkCallback-10171-thread-1-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2727239 INFO  (zkCallback-10171-thread-3-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2728135 INFO  (qtp1387007808-31124) [n:127.0.0.1:50137_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_recovery&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3662
   [junit4]   2> 2728144 INFO  (qtp1387007808-31127) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:50138/solr/tlog_replica_test_recovery_shard1_replica_t2/&wt=javabin&version=2}{add=[3 (1591188488318877696), 4 (1591188488320974848)]} 0 1
   [junit4]   2> 2728144 INFO  (qtp469824676-31132) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={_stateVer_=tlog_replica_test_recovery:5&wt=javabin&version=2}{add=[3 (1591188488318877696), 4 (1591188488320974848)]} 0 4
   [junit4]   2> 2728146 INFO  (qtp1387007808-31128) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:50138/solr/tlog_replica_test_recovery_shard1_replica_t2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
   [junit4]   2> 2728147 INFO  (qtp469824676-31136) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1591188488326217728,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2728147 INFO  (qtp469824676-31136) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@78f4edc commitCommandVersion:1591188488326217728
   [junit4]   2> 2728158 INFO  (qtp469824676-31136) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1ab7c2bb[tlog_replica_test_recovery_shard1_replica_t2] main]
   [junit4]   2> 2728158 INFO  (qtp469824676-31136) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2728160 INFO  (searcherExecutor-10299-thread-1-processing-n:127.0.0.1:50138_solr x:tlog_replica_test_recovery_shard1_replica_t2 s:shard1 c:tlog_replica_test_recovery r:core_node4) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t2] Registered new searcher Searcher@1ab7c2bb[tlog_replica_test_recovery_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C2)))}
   [junit4]   2> 2728160 INFO  (qtp469824676-31136) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:50138/solr/tlog_replica_test_recovery_shard1_replica_t2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 13
   [junit4]   2> 2728161 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={_stateVer_=tlog_replica_test_recovery:5&commit=true&wt=javabin&version=2}{commit=} 0 15
   [junit4]   2> 2728165 INFO  (qtp1387007808-31127) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:50138/solr/tlog_replica_test_recovery_shard1_replica_t2/&wt=javabin&version=2}{add=[5 (1591188488342994944)]} 0 0
   [junit4]   2> 2728166 INFO  (qtp469824676-31137) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5 (1591188488342994944)]} 0 3
   [junit4]   2> 2728167 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.c.ChaosMonkey monkey: kill jetty! 50137
   [junit4]   2> 2728168 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4667b60{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2728168 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1230440700
   [junit4]   2> 2728169 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2728169 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1a83874: rootName = solr_50137, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@42ce66e
   [junit4]   2> 2728181 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2728181 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@16842cc4: rootName = solr_50137, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@75099da1
   [junit4]   2> 2728190 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2728190 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@39d30d60: rootName = solr_50137, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@7aae97a4
   [junit4]   2> 2728192 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50137_solr
   [junit4]   2> 2728194 INFO  (zkCallback-10181-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2728194 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2728194 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2728194 INFO  (zkCallback-10171-thread-3-processing-n:127.0.0.1:50137_solr) [n:127.0.0.1:50137_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2728196 INFO  (coreCloseExecutor-10311-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_recovery_shard1_replica_t1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3535590e
   [junit4]   2> 2728196 INFO  (coreCloseExecutor-10311-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_recovery.shard1.replica_t1, tag=892688654
   [junit4]   2> 2728196 INFO  (coreCloseExecutor-10311-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3abce669: rootName = solr_50137, domain = solr.core.tlog_replica_test_recovery.shard1.replica_t1, service url = null, agent id = null] for registry solr.core.tlog_replica_test_recovery.shard1.replica_t1 / com.codahale.metrics.MetricRegistry@18f41f8d
   [junit4]   2> 2728205 INFO  (coreCloseExecutor-10311-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_recovery.shard1.leader, tag=892688654
   [junit4]   2> 2728207 INFO  (coreCloseExecutor-10311-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
   [junit4]   2> 2728207 INFO  (coreCloseExecutor-10311-thread-1) [n:127.0.0.1:50137_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@120c74fd commitCommandVersion:0
   [junit4]   2> 2728210 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.c.Overseer Overseer (id=72122435697967110-127.0.0.1:50137_solr-n_0000000000) closing
   [junit4]   2> 2728210 INFO  (OverseerStateUpdate-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50137_solr
   [junit4]   2> 2728211 WARN  (OverseerAutoScalingTriggerThread-72122435697967110-127.0.0.1:50137_solr-n_0000000000) [n:127.0.0.1:50137_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 2728213 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50138_solr
   [junit4]   2> 2728215 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@332bcb7a{/solr,null,UNAVAILABLE}
   [junit4]   2> 2728215 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 2728216 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.Overseer Overseer (id=72122435697967109-127.0.0.1:50138_solr-n_0000000001) starting
   [junit4]   2> 2728216 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 50137
   [junit4]   2> 2728223 INFO  (qtp469824676-31134) [n:127.0.0.1:50138_solr c:tlog_replica_test_recovery s:shard1 r:core_node4 x:tlog_replica_test_recovery_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_recovery_shard1_replica_t2]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6 (1591188488405909504)]} 0 0
   [junit4]   2> 2728223 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.a.s.c.ChaosMonkey monkey: starting jetty! 50137
   [junit4]   2> 2728223 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2728227 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2728227 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2728227 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2728230 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@715a144d{/solr,null,AVAILABLE}
   [junit4]   2> 2728232 INFO  (zkCallback-10169-thread-1-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2728232 INFO  (zkCallback-10169-thread-2-processing-n:127.0.0.1:50138_solr) [n:127.0.0.1:50138_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_recovery/state.json] for collection [tlog_replica_test_recovery] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2728232 INFO  (TEST-TestTlogReplica.testRecovery-seed#[624230E011B6037C]) [    ] o.e.j

[...truncated too long message...]

00000000
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72122470840729611-core_node8-n_0000000003
   [junit4]   2> 2 /solr/overseer_elect/election/72122470840729629-127.0.0.1:52119_solr-n_0000000008
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72122470840729605-core_node10-n_0000000004
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72122470840729614-core_node6-n_0000000002
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 22 /solr/live_nodes
   [junit4]   2> 22 /solr/collections
   [junit4]   2> 6 /solr/overseer/queue-work
   [junit4]   2> 6 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 6 /solr/overseer/queue
   [junit4]   2> 6 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/autoscaling/events/node_lost_triggerMR
   [junit4]   2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_624230E011B6037C-001
   [junit4]   2> Feb 01, 2018 9:12:09 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=8, maxDocsPerChunk=814, blockSize=742), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=8, blockSize=742)), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3cf09b88), locale=pt-GQ, timezone=Asia/Krasnoyarsk
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=1,free=274232032,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [AddSchemaFieldsUpdateProcessorFactoryTest, TestSizeLimitedDistributedMap, ReplicationFactorTest, DebugComponentTest, TestMinMaxOnMultiValuedField, SimpleFacetsTest, TestDefaultStatsCache, ChaosMonkeySafeLeaderWithPullReplicasTest, ClassificationUpdateProcessorIntegrationTest, SolrCmdDistributorTest, SolrCoreCheckLockOnStartupTest, DisMaxRequestHandlerTest, TestLargeCluster, SaslZkACLProviderTest, DocValuesMissingTest, SolrMetricReporterTest, SpellCheckComponentTest, TestSegmentSorting, TestComplexPhraseLeadingWildcard, TestPivotHelperCode, TestCloudNestedDocsSort, CloneFieldUpdateProcessorFactoryTest, TestNoOpRegenerator, BooleanFieldTest, ResponseLogComponentTest, TestSimDistributedQueue, TestIndexingPerformance, TestRestoreCore, TestSolrCoreParser, CustomHighlightComponentTest, BadComponentTest, TestPerFieldSimilarityWithDefaultOverride, SuggesterTest, TestSearcherReuse, EchoParamsTest, TestRequestStatusCollectionAPI, BadIndexSchemaTest, ChaosMonkeySafeLeaderTest, ReplaceNodeNoTargetTest, BasicAuthStandaloneTest, DistributedFacetPivotWhiteBoxTest, CheckHdfsIndexTest, TestSolrQueryResponse, ClusterStateUpdateTest, FacetPivotSmallTest, TestWordDelimiterFilterFactory, IgnoreCommitOptimizeUpdateProcessorFactoryTest, ZkControllerTest, TestLMJelinekMercerSimilarityFactory, JsonLoaderTest, TestDocTermOrds, TestQuerySenderListener, TestRawTransformer, OverseerTaskQueueTest, ScriptEngineTest, TestCloudJSONFacetJoinDomain, TermVectorComponentTest, FullSolrCloudDistribCmdsTest, MetricUtilsTest, TestManagedSchemaThreadSafety, TestCollectionsAPIViaSolrCloudCluster, BaseCdcrDistributedZkTest, TestCoreBackup, ReturnFieldsTest, TestLegacyNumericRangeQueryBuilder, TestSubQueryTransformerDistrib, ZkStateWriterTest, TestSearchPerf, HdfsTlogReplayBufferedWhileIndexingTest, TestLeaderElectionWithEmptyReplica, TestManagedStopFilterFactory, UpdateParamsTest, TestFieldCacheReopen, DeleteReplicaTest, TestRandomFaceting, TestDynamicFieldCollectionResource, SpatialRPTFieldTypeTest, TestComponentsName, PreAnalyzedUpdateProcessorTest, TestHashQParserPlugin, OutOfBoxZkACLAndCredentialsProvidersTest, TestSolrCloudWithHadoopAuthPlugin, TestExceedMaxTermLength, TestLRUStatsCache, TestStressVersions, DistributedIntervalFacetingTest, TestDownShardTolerantSearch, TestComputePlanAction, ConnectionManagerTest, TestMiniSolrCloudClusterSSL, AutoAddReplicasPlanActionTest, TestJettySolrRunner, AsyncCallRequestStatusResponseTest, TestDistributedMissingSort, TestSolrIndexConfig, TestExactSharedStatsCache, TestDFISimilarityFactory, RankQueryTest, LegacyCloudClusterPropTest, TestDistributedGrouping, MinimalSchemaTest, SolrJmxReporterCloudTest, TestDeleteCollectionOnDownNodes, TestCloudDeleteByQuery, SuggesterWFSTTest, TestSerializedLuceneMatchVersion, SuggestComponentTest, SliceStateTest, TestDocumentBuilder, SchemaApiFailureTest, PrimitiveFieldTypeTest, AtomicUpdatesTest, AnalysisErrorHandlingTest, QueryParsingTest, JSONWriterTest, RollingRestartTest, PrimUtilsTest, TestDFRSimilarityFactory, TestStressInPlaceUpdates, TestLocalFSCloudBackupRestore, PeerSyncTest, TestLeaderInitiatedRecoveryThread, HdfsAutoAddReplicasIntegrationTest, TestExtendedDismaxParser, CSVRequestHandlerTest, JavabinLoaderTest, TestRTGBase, TestFileDictionaryLookup, NumberUtilsTest, TestObjectReleaseTracker, TestFieldCollectionResource, TestFilteredDocIdSet, JvmMetricsTest, TlogReplayBufferedWhileIndexingTest, CoreAdminRequestStatusTest, FieldAnalysisRequestHandlerTest, UUIDUpdateProcessorFallbackTest, TestSubQueryTransformer, CdcrVersionReplicationTest, UpdateLogTest, UUIDFieldTest, SystemLogListenerTest, TestLuceneMatchVersion, TestNumericTerms32, TestPushWriter, TestAuthorizationFramework, QueryResultKeyTest, HdfsWriteToMultipleCollectionsTest, DataDrivenBlockJoinTest, TestSchemaNameResource, DeleteLastCustomShardedReplicaTest, TestGraphMLResponseWriter, TestNumericTokenStream, SolrCoreMetricManagerTest, TestRecoveryHdfs, CoreAdminHandlerTest, HdfsCollectionsAPIDistributedZkTest, TestPullReplicaErrorHandling, TestReplicationHandlerBackup, PropertiesRequestHandlerTest, SimplePostToolTest, ClusterStateTest, TestAnalyzedSuggestions, HdfsNNFailoverTest, TestDelegationWithHadoopAuth, TestStressReorder, UninvertDocValuesMergePolicyTest, TestMacroExpander, ConjunctionSolrSpellCheckerTest, TestFuzzyAnalyzedSuggestions, MultiThreadedOCPTest, TestWriterPerf, HighlighterConfigTest, TestSQLHandlerNonCloud, MoveReplicaHDFSTest, TestPHPSerializedResponseWriter, TestManagedResourceStorage, PreAnalyzedFieldTest, LoggingHandlerTest, HttpPartitionTest, TestCrossCoreJoin, TestHashPartitioner, TestCodecSupport, TestExportWriter, TestSolrCoreProperties, TestRuleBasedAuthorizationPlugin, VMParamsZkACLAndCredentialsProvidersTest, SolrShardReporterTest, TestStandardQParsers, RecoveryAfterSoftCommitTest, BasicDistributedZkTest, TestReplicationHandler, TestDistributedSearch, TestJoin, DistributedTermsComponentTest, TestCoreContainer, TestSolr4Spatial, SolrCoreTest, TestGroupingSearch, TestSort, TestFiltering, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SuggesterFSTTest, DocValuesTest, SuggesterTSTTest, TestStressLucene, TestTrie, SpatialFilterTest, TestCSVLoader, NoCacheHeaderTest, TestUpdate, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, DocValuesMultiTest, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, TestQueryUtils, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, TestOmitPositions, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, TestValueSourceCache, MoreLikeThisHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, UpdateRequestProcessorFactoryTest, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, SearchHandlerTest, TestLFUCache, TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, TestStressRecovery, TestMergePolicyConfig, TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, TestBinaryField, NumericFieldsTest, TestConfig, ExternalFileFieldSortTest, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, ConcurrentCreateRoutedAliasTest, CreateCollectionCleanupTest, DeleteShardTest, DeleteStatusTest, DistribDocExpirationUpdateProcessorTest, DocValuesNotIndexedTest, HealthCheckHandlerTest, LIRRollingUpdatesTest, MigrateRouteKeyTest, MissingSegmentRecoveryTest, NodeMutatorTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerRolesTest, RemoteQueryErrorTest, ReplaceNodeTest, TestCloudPseudoReturnFields, TestCloudSearcherWarming, TestConfigSetsAPIExclusivity, TestLeaderElectionZkExpiry, TestLockTree, TestOnReconnectListenerSupport, TestPullReplica, TestStressCloudBlindAtomicUpdates, TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, TestUtilizeNode, ZkShardTermsTest, AssignTest, SimpleCollectionCreateDeleteTest, TestCollectionAPI, TestHdfsCloudBackupRestore, TestReplicaProperties, AutoAddReplicasIntegrationTest, ComputePlanActionTest, ExecutePlanActionTest, TriggerIntegrationTest]
   [junit4] Completed [625/774 (3!)] on J1 in 207.77s, 14 tests, 2 failures <<< FAILURES!

[...truncated 41399 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4419 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4419/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
        at __randomizedtesting.SeedInfo.seed([BD0380BE22963D59:3557BF648C6A50A1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:915)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory

Error Message:
expected:<5> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<0>
        at __randomizedtesting.SeedInfo.seed([BD0380BE22963D59:D0FF244398DEC25E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:274)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12093 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/init-core-data-001
   [junit4]   2> 644404 WARN  (SUITE-FullSolrCloudDistribCmdsTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=19 numCloses=19
   [junit4]   2> 644404 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 644407 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 644408 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 644408 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 644409 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 644409 INFO  (Thread-969) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 644409 INFO  (Thread-969) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 644416 ERROR (Thread-969) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 644518 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ZkTestServer start zk server on port:52625
   [junit4]   2> 644529 INFO  (zkConnectionManagerCallback-1056-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 644536 INFO  (zkConnectionManagerCallback-1058-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 644546 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 644549 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 644551 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 644553 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 644555 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 644557 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 644559 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 644561 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 644563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 644566 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 644568 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 644571 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 644861 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 644862 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 644862 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 644863 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 644863 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@30512318{/,null,AVAILABLE}
   [junit4]   2> 644865 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2681c909{HTTP/1.1,[http/1.1]}{127.0.0.1:52628}
   [junit4]   2> 644865 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server Started @649090ms
   [junit4]   2> 644865 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=52628, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/control-001/cores}
   [junit4]   2> 644865 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 644866 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 644866 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 644866 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 644866 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-01T20:22:04.227Z
   [junit4]   2> 644868 INFO  (zkConnectionManagerCallback-1060-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 644869 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 644869 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/control-001/solr.xml
   [junit4]   2> 644874 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 644874 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 644875 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 644878 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52625/solr
   [junit4]   2> 644880 INFO  (zkConnectionManagerCallback-1064-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 644883 INFO  (zkConnectionManagerCallback-1066-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 645025 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 645026 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52628_
   [junit4]   2> 645027 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.c.Overseer Overseer (id=72125118569250820-127.0.0.1:52628_-n_0000000000) starting
   [junit4]   2> 645038 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52628_
   [junit4]   2> 645040 INFO  (OverseerStateUpdate-72125118569250820-127.0.0.1:52628_-n_0000000000) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 645112 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 645118 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 645119 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 645120 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52628_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/control-001/cores
   [junit4]   2> 645142 INFO  (zkConnectionManagerCallback-1072-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 645144 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 645145 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52625/solr ready
   [junit4]   2> 645148 INFO  (qtp1293205465-4620) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:52628_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 645151 INFO  (OverseerThreadFactory-1768-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 645269 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 645269 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 645385 INFO  (zkCallback-1065-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 646339 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 646374 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 646418 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 646444 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 646444 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 646445 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 646445 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 646448 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3551283249084348]
   [junit4]   2> 646452 WARN  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 646507 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 646507 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 646509 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 646509 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 646510 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=24.95703125, floorSegmentMB=1.4736328125, forceMergeDeletesPctAllowed=17.609156860898246, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4285863542326893
   [junit4]   2> 646510 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@878d85d[control_collection_shard1_replica_n1] main]
   [junit4]   2> 646512 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 646512 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 646513 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 646513 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591231408634855424
   [junit4]   2> 646514 INFO  (searcherExecutor-1771-thread-1-processing-n:127.0.0.1:52628_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@878d85d[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 646524 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 646524 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 646524 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52628/control_collection_shard1_replica_n1/
   [junit4]   2> 646524 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 646524 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:52628/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 646524 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 646533 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52628/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 646641 INFO  (zkCallback-1065-thread-2-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 646693 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 646695 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1426
   [junit4]   2> 646703 INFO  (qtp1293205465-4620) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 646801 INFO  (zkCallback-1065-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 647155 INFO  (OverseerCollectionConfigSetProcessor-72125118569250820-127.0.0.1:52628_-n_0000000000) [n:127.0.0.1:52628_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 647708 INFO  (qtp1293205465-4620) [n:127.0.0.1:52628_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:52628_&wt=javabin&version=2} status=0 QTime=2559
   [junit4]   2> 647730 INFO  (zkConnectionManagerCallback-1076-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 647732 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 647735 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52625/solr ready
   [junit4]   2> 647735 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 647736 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 647740 INFO  (OverseerThreadFactory-1768-thread-2-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 647741 WARN  (OverseerThreadFactory-1768-thread-2-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 647958 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 647959 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=222
   [junit4]   2> 648561 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-1-001 of type NRT
   [junit4]   2> 648562 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 648563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 648563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 648563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 648564 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b9e35dd{/,null,AVAILABLE}
   [junit4]   2> 648565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@40d93a76{HTTP/1.1,[http/1.1]}{127.0.0.1:52767}
   [junit4]   2> 648565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server Started @652790ms
   [junit4]   2> 648565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=52767, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-1-001/cores}
   [junit4]   2> 648565 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 648565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 648565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 648565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 648566 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-01T20:22:07.927Z
   [junit4]   2> 648568 INFO  (zkConnectionManagerCallback-1078-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 648571 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 648571 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-1-001/solr.xml
   [junit4]   2> 648591 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 648591 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 648592 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 648598 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52625/solr
   [junit4]   2> 648603 INFO  (zkConnectionManagerCallback-1082-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 648609 INFO  (zkConnectionManagerCallback-1084-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 648650 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 648657 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 648668 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52767_
   [junit4]   2> 648671 INFO  (zkCallback-1065-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 648674 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 648675 INFO  (zkCallback-1075-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 648808 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 648828 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 648828 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 648832 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52767_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-1-001/cores
   [junit4]   2> 648916 INFO  (qtp1293205465-4622) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:52767_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 648921 INFO  (OverseerCollectionConfigSetProcessor-72125118569250820-127.0.0.1:52628_-n_0000000000) [n:127.0.0.1:52628_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 648922 INFO  (OverseerThreadFactory-1768-thread-3-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:52767_ for creating new replica
   [junit4]   2> 648931 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 648932 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 649047 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 649976 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 649993 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema [collection1_shard2_replica_n61] Schema name=test
   [junit4]   2> 650088 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 650122 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n61' using configuration from collection collection1, trusted=true
   [junit4]   2> 650123 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n61' (registry 'solr.core.collection1.shard2.replica_n61') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 650123 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 650123 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [[collection1_shard2_replica_n61] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-1-001/cores/collection1_shard2_replica_n61], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-1-001/cores/collection1_shard2_replica_n61/data/]
   [junit4]   2> 650128 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3551283249084348]
   [junit4]   2> 650134 WARN  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 650250 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 650250 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 650253 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 650253 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 650256 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=24.95703125, floorSegmentMB=1.4736328125, forceMergeDeletesPctAllowed=17.609156860898246, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4285863542326893
   [junit4]   2> 650256 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.SolrIndexSearcher Opening [Searcher@1ead49c7[collection1_shard2_replica_n61] main]
   [junit4]   2> 650258 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 650259 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 650260 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 650261 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591231412564918272
   [junit4]   2> 650262 INFO  (searcherExecutor-1782-thread-1-processing-n:127.0.0.1:52767_ x:collection1_shard2_replica_n61 s:shard2 c:collection1) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [collection1_shard2_replica_n61] Registered new searcher Searcher@1ead49c7[collection1_shard2_replica_n61] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 650274 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 650274 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 650274 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52767/collection1_shard2_replica_n61/
   [junit4]   2> 650275 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 650275 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy http://127.0.0.1:52767/collection1_shard2_replica_n61/ has no replicas
   [junit4]   2> 650275 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 650281 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52767/collection1_shard2_replica_n61/ shard2
   [junit4]   2> 650383 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 650395 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 650399 INFO  (qtp485647419-4668) [n:127.0.0.1:52767_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1467
   [junit4]   2> 650403 INFO  (qtp1293205465-4622) [n:127.0.0.1:52628_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:52767_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1486
   [junit4]   2> 650500 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 650922 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-2-001 of type NRT
   [junit4]   2> 650922 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 650924 INFO  (OverseerCollectionConfigSetProcessor-72125118569250820-127.0.0.1:52628_-n_0000000000) [n:127.0.0.1:52628_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 650924 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 650924 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 650925 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 650925 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55f39cdf{/,null,AVAILABLE}
   [junit4]   2> 650926 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7589fdba{HTTP/1.1,[http/1.1]}{127.0.0.1:52863}
   [junit4]   2> 650926 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server Started @655151ms
   [junit4]   2> 650926 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=52863, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-2-001/cores}
   [junit4]   2> 650927 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 650929 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 650929 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 650929 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 650929 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-01T20:22:10.290Z
   [junit4]   2> 650931 INFO  (zkConnectionManagerCallback-1088-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 650933 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 650933 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-2-001/solr.xml
   [junit4]   2> 650944 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 650944 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 650945 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 650954 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52625/solr
   [junit4]   2> 650956 INFO  (zkConnectionManagerCallback-1092-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 650961 INFO  (zkConnectionManagerCallback-1094-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 650972 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 650978 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 650983 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52863_
   [junit4]   2> 650985 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 650985 INFO  (zkCallback-1065-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 650985 INFO  (zkCallback-1075-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 650986 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 651109 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 651122 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 651122 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 651125 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52863_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-2-001/cores
   [junit4]   2> 651170 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:52863_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 651178 INFO  (OverseerThreadFactory-1768-thread-4-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:52863_ for creating new replica
   [junit4]   2> 651184 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 651186 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 651295 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 651295 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 652219 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 652234 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema [collection1_shard3_replica_n63] Schema name=test
   [junit4]   2> 652367 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 652407 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_n63' using configuration from collection collection1, trusted=true
   [junit4]   2> 652408 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_n63' (registry 'solr.core.collection1.shard3.replica_n63') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 652408 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 652408 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [[collection1_shard3_replica_n63] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-2-001/cores/collection1_shard3_replica_n63], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-2-001/cores/collection1_shard3_replica_n63/data/]
   [junit4]   2> 652411 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3551283249084348]
   [junit4]   2> 652417 WARN  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 652493 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 652493 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 652494 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 652494 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 652495 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=24.95703125, floorSegmentMB=1.4736328125, forceMergeDeletesPctAllowed=17.609156860898246, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4285863542326893
   [junit4]   2> 652496 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.SolrIndexSearcher Opening [Searcher@1dbcb7a8[collection1_shard3_replica_n63] main]
   [junit4]   2> 652497 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 652498 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 652498 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 652498 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591231414910582784
   [junit4]   2> 652499 INFO  (searcherExecutor-1793-thread-1-processing-n:127.0.0.1:52863_ x:collection1_shard3_replica_n63 s:shard3 c:collection1) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [collection1_shard3_replica_n63] Registered new searcher Searcher@1dbcb7a8[collection1_shard3_replica_n63] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 652506 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 652506 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 652506 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52863/collection1_shard3_replica_n63/
   [junit4]   2> 652506 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 652506 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy http://127.0.0.1:52863/collection1_shard3_replica_n63/ has no replicas
   [junit4]   2> 652506 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 652511 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52863/collection1_shard3_replica_n63/ shard3
   [junit4]   2> 652614 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 652614 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 652615 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 652620 INFO  (qtp825921177-4700) [n:127.0.0.1:52863_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1436
   [junit4]   2> 652624 INFO  (qtp1293205465-4624) [n:127.0.0.1:52628_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:52863_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2} status=0 QTime=1454
   [junit4]   2> 652723 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 652723 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 653104 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-3-001 of type NRT
   [junit4]   2> 653104 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 653105 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 653106 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 653106 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 653106 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@115a0885{/,null,AVAILABLE}
   [junit4]   2> 653107 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@72d314d9{HTTP/1.1,[http/1.1]}{127.0.0.1:52992}
   [junit4]   2> 653108 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.e.j.s.Server Started @657333ms
   [junit4]   2> 653108 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=52992, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-3-001/cores}
   [junit4]   2> 653108 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 653109 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 653109 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 653109 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 653109 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-01T20:22:12.470Z
   [junit4]   2> 653112 INFO  (zkConnectionManagerCallback-1098-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 653114 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 653114 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-3-001/solr.xml
   [junit4]   2> 653126 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 653126 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 653165 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 653169 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52625/solr
   [junit4]   2> 653171 INFO  (zkConnectionManagerCallback-1102-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 653176 INFO  (zkConnectionManagerCallback-1104-thread-1-processing-n:127.0.0.1:52992_) [n:127.0.0.1:52992_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 653179 INFO  (OverseerCollectionConfigSetProcessor-72125118569250820-127.0.0.1:52628_-n_0000000000) [n:127.0.0.1:52628_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 653188 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 653193 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 653196 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52992_
   [junit4]   2> 653199 INFO  (zkCallback-1075-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 653199 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 653199 INFO  (zkCallback-1065-thread-1-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 653199 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 653200 INFO  (zkCallback-1103-thread-1-processing-n:127.0.0.1:52992_) [n:127.0.0.1:52992_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 653367 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 653379 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 653379 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 653384 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[BD0380BE22963D59]) [n:127.0.0.1:52992_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-3-001/cores
   [junit4]   2> 653461 INFO  (qtp1293205465-4622) [n:127.0.0.1:52628_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:52992_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 653466 INFO  (OverseerThreadFactory-1768-thread-5-processing-n:127.0.0.1:52628_) [n:127.0.0.1:52628_    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:52992_ for creating new replica
   [junit4]   2> 653475 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n65&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 653476 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 653586 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 653586 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 653586 INFO  (zkCallback-1103-thread-1-processing-n:127.0.0.1:52992_) [n:127.0.0.1:52992_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 654500 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 654509 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema [collection1_shard1_replica_n65] Schema name=test
   [junit4]   2> 654620 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 654647 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n65' using configuration from collection collection1, trusted=true
   [junit4]   2> 654647 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n65' (registry 'solr.core.collection1.shard1.replica_n65') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f526f48
   [junit4]   2> 654648 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 654648 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore [[collection1_shard1_replica_n65] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-3-001/cores/collection1_shard1_replica_n65], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_BD0380BE22963D59-001/shard-3-001/cores/collection1_shard1_replica_n65/data/]
   [junit4]   2> 654662 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3551283249084348]
   [junit4]   2> 654703 WARN  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 654764 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 654764 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 654771 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 654771 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 654783 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=24.95703125, floorSegmentMB=1.4736328125, forceMergeDeletesPctAllowed=17.609156860898246, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4285863542326893
   [junit4]   2> 654796 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.SolrIndexSearcher Opening [Searcher@1f05ba99[collection1_shard1_replica_n65] main]
   [junit4]   2> 654797 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 654798 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 654798 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 654799 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1591231417323356160
   [junit4]   2> 654799 INFO  (searcherExecutor-1804-thread-1-processing-n:127.0.0.1:52992_ x:collection1_shard1_replica_n65 s:shard1 c:collection1) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore [collection1_shard1_replica_n65] Registered new searcher Searcher@1f05ba99[collection1_shard1_replica_n65] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 654806 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 654806 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 654806 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52992/collection1_shard1_replica_n65/
   [junit4]   2> 654806 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 654806 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy http://127.0.0.1:52992/collection1_shard1_replica_n65/ has no replicas
   [junit4]   2> 654806 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 654810 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52992/collection1_shard1_replica_n65/ shard1
   [junit4]   2> 654913 INFO  (zkCallback-1093-thread-1-processing-n:127.0.0.1:52863_) [n:127.0.0.1:52863_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 654913 INFO  (zkCallback-1103-thread-1-processing-n:127.0.0.1:52992_) [n:127.0.0.1:52992_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 654913 INFO  (zkCallback-1083-thread-1-processing-n:127.0.0.1:52767_) [n:127.0.0.1:52767_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 654927 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 654930 INFO  (qtp1361686455-4731) [n:127.0.0.1:52992_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection

[...truncated too long message...]

AILABLE}
   [junit4]   2> 3802736 INFO  (jetty-closer-10287-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 3804236 INFO  (jetty-closer-10287-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@18376e66{/solr,null,UNAVAILABLE}
   [junit4]   2> 3804236 INFO  (jetty-closer-10287-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 3804236 INFO  (jetty-closer-10287-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@77e447dd{/solr,null,UNAVAILABLE}
   [junit4]   2> 3804236 INFO  (jetty-closer-10287-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 3804238 ERROR (SUITE-AutoscalingHistoryHandlerTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3804238 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[BD0380BE22963D59]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50776 50776
   [junit4]   2> 3804276 INFO  (Thread-5494) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50776 50776
   [junit4]   2> 3804595 WARN  (Thread-5494) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 21 /solr/collections/.system/terms/shard1
   [junit4]   2> 18 /solr/collections/AutoscalingHistoryHandlerTest_collection/terms/shard1
   [junit4]   2> 13 /solr/aliases.json
   [junit4]   2> 13 /solr/clusterprops.json
   [junit4]   2> 5 /solr/configs/.system
   [junit4]   2> 5 /solr/configs/.system/managed-schema
   [junit4]   2> 4 /solr/security.json
   [junit4]   2> 4 /solr/configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/managed-schema
   [junit4]   2> 4 /solr/configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 60 /solr/collections/.system/state.json
   [junit4]   2> 40 /solr/collections/AutoscalingHistoryHandlerTest_collection/state.json
   [junit4]   2> 13 /solr/clusterstate.json
   [junit4]   2> 2 /solr/collections/.system/leader_elect/shard1/election/72125323097473033-core_node6-n_0000000000
   [junit4]   2> 2 /solr/collections/AutoscalingHistoryHandlerTest_collection/leader_elect/shard1/election/72125323097473031-core_node3-n_0000000000
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 13 /solr/live_nodes
   [junit4]   2> 13 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/autoscaling/events/AutoscalingHistoryHandlerTest_node_lost_trigger
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_BD0380BE22963D59-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@108fc5b5), locale=ar-YE, timezone=Greenwich
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=93306776,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [OpenCloseCoreStressTest, TestEmbeddedSolrServerAdminHandler, UUIDUpdateProcessorFallbackTest, UpdateLogTest, TestRecovery, TestSolrXml, SolrMetricsIntegrationTest, AnalysisErrorHandlingTest, SpellingQueryConverterTest, TestSolrConfigHandlerCloud, TestNumericTerms32, TestXIncludeConfig, TestObjectReleaseTracker, SuggesterTSTTest, TestSearcherReuse, RecoveryZkTest, TestSchemalessBufferedUpdates, MBeansHandlerTest, LeaderElectionIntegrationTest, TestCloudSchemaless, TestCoreAdminApis, HdfsRecoverLeaseTest, OutputWriterTest, TestBulkSchemaConcurrent, TestJsonFacetRefinement, TestFilteredDocIdSet, PreAnalyzedUpdateProcessorTest, DistributedMLTComponentTest, TestRestManager, EnumFieldTest, DistributedVersionInfoTest, TestPolicyCloud, AddReplicaTest, CoreAdminCreateDiscoverTest, DirectUpdateHandlerOptimizeTest, TestFieldResource, BigEndianAscendingWordDeserializerTest, RequiredFieldsTest, DistributedSuggestComponentTest, TestTrackingShardHandlerFactory, TestZkChroot, DistributedFacetPivotLongTailTest, TestCharFilters, TestJavabinTupleStreamParser, ImplicitSnitchTest, CacheHeaderTest, TestSimpleTextCodec, TestJsonRequest, TestChildDocTransformer, TestInitParams, TestSSLRandomization, TestPayloadCheckQParserPlugin, BitVectorTest, TestLegacyField, SolrCoreTest, SolrIndexConfigTest, NodeMutatorTest, TestReload, TestCustomStream, HdfsChaosMonkeySafeLeaderTest, MoreLikeThisHandlerTest, TestSolrCloudWithKerberosAlt, MetricUtilsTest, DistributedQueryComponentCustomSortTest, LIRRollingUpdatesTest, PeerSyncReplicationTest, TestClusterProperties, ShardRoutingCustomTest, TestUniqueKeyFieldResource, AlternateDirectoryTest, StatsComponentTest, XmlUpdateRequestHandlerTest, TestRandomFlRTGCloud, MultiTermTest, TestFunctionQuery, TestSQLHandler, TestBM25SimilarityFactory, DistribCursorPagingTest, TestGeoJSONResponseWriter, BasicAuthStandaloneTest, TestRetrieveFieldsOptimizer, HdfsUnloadDistributedZkTest, TriLevelCompositeIdRoutingTest, TestConfigSets, StatsReloadRaceTest, TestAddFieldRealTimeGet, BasicDistributedZkTest, TestCryptoKeys, BinaryUpdateRequestHandlerTest, ConjunctionSolrSpellCheckerTest, SpatialFilterTest, SuggesterFSTTest, RankQueryTest, DistributedQueryElevationComponentTest, DistributedFacetPivotSmallTest, AssignBackwardCompatibilityTest, TestMultiValuedNumericRangeQuery, MoveReplicaTest, TestFastOutputStream, LargeFieldTest, TestDelegationWithHadoopAuth, TestCloudPivotFacet, HdfsDirectoryTest, AutoAddReplicasIntegrationTest, TestLazyCores, TestElisionMultitermQuery, TestQuerySenderNoQuery, DirectSolrSpellCheckerTest, TestOmitPositions, TestRealTimeGet, TestUtils, ZkFailoverTest, TestReplicationHandler, TestEmbeddedSolrServerSchemaAPI, HDFSCollectionsAPITest, TestJmxIntegration, RestartWhileUpdatingTest, SolrInfoBeanTest, AssignTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestBackupRepositoryFactory, TestHttpShardHandlerFactory, SolrMetricManagerTest, HighlighterTest, TestMinMaxOnMultiValuedField, TestPointFields, TestReloadAndDeleteDocs, TestCloudNestedDocsSort, ClassificationUpdateProcessorIntegrationTest, TestIBSimilarityFactory, TestReqParamsAPI, TimeRoutedAliasUpdateProcessorTest, SynonymTokenizerTest, HdfsThreadLeakTest, InfixSuggestersTest, TestSystemIdResolver, TestImplicitCoreProperties, TestReplicaProperties, ConvertedLegacyTest, TestGenericDistributedQueue, MigrateRouteKeyTest, SchemaVersionSpecificBehaviorTest, TestPhraseSuggestions, TestStandardQParsers, MetricsConfigTest, BlockJoinFacetSimpleTest, SolrGraphiteReporterTest, UnloadDistributedZkTest, TestNumericRangeQuery64, TestConfigReload, TestMacros, FileUtilsTest, TestSolrCLIRunExample, TestCustomDocTransformer, DirectSolrConnectionTest, MetricsHandlerTest, FieldMutatingUpdateProcessorTest, SolrJmxReporterTest, TestFoldingMultitermQuery, CdcrRequestHandlerTest, TestHdfsBackupRestoreCore, TestPolicyCloud, TestLockTree, BlockDirectoryTest, TestPerFieldSimilarityWithDefaultOverride, FastVectorHighlighterTest, SolrCloudExampleTest, TestMissingGroups, TestSystemCollAutoCreate, TestCollectionAPI, CreateCollectionCleanupTest, SharedFSAutoReplicaFailoverTest, IndexSchemaRuntimeFieldTest, CurrencyRangeFacetCloudTest, DateMathParserTest, TestStreamBody, UtilsToolTest, BadCopyFieldTest, CopyFieldTest, RemoteQueryErrorTest, TestQueryTypes, NodeAddedTriggerTest, TestPostingsSolrHighlighter, TestStressCloudBlindAtomicUpdates, CollectionsAPISolrJTest, TestSuggestSpellingConverter, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, TestRTGBase, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ConnectionReuseTest, AsyncCallRequestStatusResponseTest, CloudExitableDirectoryReaderTest, ConcurrentCreateRoutedAliasTest, CreateRoutedAliasTest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, DocValuesNotIndexedTest, ForceLeaderTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerStatusTest, OverseerTaskQueueTest, ReplaceNodeNoTargetTest, ReplaceNodeTest, SSLMigrationTest, SaslZkACLProviderTest, TestCloudDeleteByQuery, TestCloudRecovery, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestOnReconnectListenerSupport, TestRebalanceLeaders, TestShortCircuitedRequests, TestSkipOverseerOperations, TestSolrCloudWithDelegationTokens, TestStressLiveNodes, TestTlogReplica, CustomCollectionTest, HdfsCollectionsAPIDistributedZkTest, ShardSplitTest, CdcrReplicationDistributedZkTest, HdfsBasicDistributedZkTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsNNFailoverTest, HdfsRecoveryZkTest, HdfsRestartWhileUpdatingTest, HdfsTlogReplayBufferedWhileIndexingTest, StressHdfsTest, TestClusterStateMutator, ZkStateReaderTest, ZkStateWriterTest, RuleEngineTest, RulesTest, TestShardHandlerFactory, TestSolrCloudSnapshots, TestReplicationHandlerBackup, TestRestoreCore, V2StandaloneTest, AutoscalingHistoryHandlerTest]
   [junit4] Completed [639/775 (2!)] on J0 in 38.77s, 1 test, 1 failure <<< FAILURES!

[...truncated 48551 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4420 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4420/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([5063219FE1E594E0:D85EA8E0DB25754D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState(TriggerIntegrationTest.java:426)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12534 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> 1269654 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/init-core-data-001
   [junit4]   2> 1269655 WARN  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 1269655 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1269658 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1269659 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/tempDir-001
   [junit4]   2> 1269659 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1269660 INFO  (Thread-2613) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1269660 INFO  (Thread-2613) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1269662 ERROR (Thread-2613) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1269760 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:65070
   [junit4]   2> 1269772 INFO  (zkConnectionManagerCallback-2629-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269778 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1269779 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1269779 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1269779 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1269780 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@93518aa{/solr,null,AVAILABLE}
   [junit4]   2> 1269780 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@713e9471{HTTP/1.1,[http/1.1]}{127.0.0.1:65072}
   [junit4]   2> 1269780 INFO  (jetty-launcher-2626-thread-2) [    ] o.e.j.s.Server Started @1275334ms
   [junit4]   2> 1269780 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65072}
   [junit4]   2> 1269780 ERROR (jetty-launcher-2626-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1269781 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1269781 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1269781 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1269781 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T08:17:22.381Z
   [junit4]   2> 1269783 INFO  (zkConnectionManagerCallback-2631-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269784 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1269785 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1269788 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1269788 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1269788 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1269788 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18e6eab{/solr,null,AVAILABLE}
   [junit4]   2> 1269788 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1dce3a24{HTTP/1.1,[http/1.1]}{127.0.0.1:65074}
   [junit4]   2> 1269789 INFO  (jetty-launcher-2626-thread-1) [    ] o.e.j.s.Server Started @1275343ms
   [junit4]   2> 1269789 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65074}
   [junit4]   2> 1269789 ERROR (jetty-launcher-2626-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1269789 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1269789 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1269789 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1269789 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T08:17:22.389Z
   [junit4]   2> 1269793 INFO  (zkConnectionManagerCallback-2633-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269794 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1269800 INFO  (jetty-launcher-2626-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1269802 INFO  (zkConnectionManagerCallback-2637-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269804 INFO  (jetty-launcher-2626-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1269808 INFO  (zkConnectionManagerCallback-2641-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269809 INFO  (zkConnectionManagerCallback-2643-thread-1-processing-n:127.0.0.1:65072_solr) [n:127.0.0.1:65072_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269812 INFO  (zkConnectionManagerCallback-2645-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1269985 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1269985 INFO  (jetty-launcher-2626-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1269986 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65072_solr
   [junit4]   2> 1269987 INFO  (jetty-launcher-2626-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65074_solr
   [junit4]   2> 1269987 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.c.Overseer Overseer (id=72127931275608069-127.0.0.1:65072_solr-n_0000000000) starting
   [junit4]   2> 1269990 INFO  (zkCallback-2642-thread-1-processing-n:127.0.0.1:65072_solr) [n:127.0.0.1:65072_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1269991 INFO  (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1269999 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1270001 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1270002 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65074_solr]
   [junit4]   2> 1270002 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1270002 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1270002 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65072_solr
   [junit4]   2> 1270007 INFO  (zkCallback-2642-thread-1-processing-n:127.0.0.1:65072_solr) [n:127.0.0.1:65072_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1270008 INFO  (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1270013 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1270014 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1270014 DEBUG (ScheduledTrigger-4593-thread-1) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1270107 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65072.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270110 INFO  (jetty-launcher-2626-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65074.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270115 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65072.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270116 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65072.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270116 INFO  (jetty-launcher-2626-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65074.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270116 INFO  (jetty-launcher-2626-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65074.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270117 INFO  (jetty-launcher-2626-thread-2) [n:127.0.0.1:65072_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/tempDir-001/node2/.
   [junit4]   2> 1270117 INFO  (jetty-launcher-2626-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/tempDir-001/node1/.
   [junit4]   2> 1270162 INFO  (zkConnectionManagerCallback-2651-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1270165 INFO  (zkConnectionManagerCallback-2655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1270167 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1270169 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:65070/solr ready
   [junit4]   2> 1270189 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 1270189 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 1270190 DEBUG (zkCallback-2642-thread-1-processing-n:127.0.0.1:65072_solr) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1270190 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1270192 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1270193 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1270194 DEBUG (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1270196 INFO  (qtp1247829508-11731) [n:127.0.0.1:65072_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1270203 INFO  (qtp1247829508-11731) [n:127.0.0.1:65072_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 1270204 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1270205 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1270205 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1270205 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1270206 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3c89523c{/solr,null,AVAILABLE}
   [junit4]   2> 1270206 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1d90551a{HTTP/1.1,[http/1.1]}{127.0.0.1:65083}
   [junit4]   2> 1270206 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.Server Started @1275760ms
   [junit4]   2> 1270206 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65083}
   [junit4]   2> 1270206 ERROR (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1270206 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1270207 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1270207 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1270207 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T08:17:22.807Z
   [junit4]   2> 1270209 INFO  (zkConnectionManagerCallback-2657-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1270210 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1270220 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1270222 INFO  (zkConnectionManagerCallback-2661-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1270224 INFO  (zkConnectionManagerCallback-2663-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1270241 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1270245 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1270249 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65083_solr
   [junit4]   2> 1270250 INFO  (zkCallback-2642-thread-1-processing-n:127.0.0.1:65072_solr) [n:127.0.0.1:65072_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1270251 INFO  (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1270251 INFO  (zkCallback-2662-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1270252 INFO  (zkCallback-2654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1270330 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65083.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270339 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65083.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270339 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65083.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1270341 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65083_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/tempDir-001/node3/.
   [junit4]   2> 1270393 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== KILL OVERSEER 1
   [junit4]   2> 1270395 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@713e9471{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1270395 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=568390547
   [junit4]   2> 1270395 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1270395 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7c6579c9: rootName = solr_65072, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@171ddaec
   [junit4]   2> 1270407 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1270407 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@8d307ca: rootName = solr_65072, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@a1c06b6
   [junit4]   2> 1270410 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1270410 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7b09d6aa: rootName = solr_65072, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@701e731c
   [junit4]   2> 1270412 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:65072_solr
   [junit4]   2> 1270413 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1270414 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1270414 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.Overseer Overseer (id=72127931275608069-127.0.0.1:65072_solr-n_0000000000) closing
   [junit4]   2> 1270415 INFO  (zkCallback-2642-thread-1-processing-n:127.0.0.1:65072_solr) [n:127.0.0.1:65072_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1270415 INFO  (zkCallback-2662-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1270415 INFO  (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1270415 INFO  (OverseerStateUpdate-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:65072_solr
   [junit4]   2> 1270415 INFO  (zkCallback-2654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1270416 WARN  (OverseerAutoScalingTriggerThread-72127931275608069-127.0.0.1:65072_solr-n_0000000000) [n:127.0.0.1:65072_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1270416 DEBUG (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1270416 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@93518aa{/solr,null,UNAVAILABLE}
   [junit4]   2> 1270421 INFO  (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65074_solr
   [junit4]   2> 1270421 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1270423 INFO  (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.Overseer Overseer (id=72127931275608070-127.0.0.1:65074_solr-n_0000000001) starting
   [junit4]   2> 1270432 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1270433 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1270434 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65074_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1270434 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion -1
   [junit4]   2> 1270434 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1270437 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1270438 DEBUG (ScheduledTrigger-4608-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1270438 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1271449 DEBUG (ScheduledTrigger-4608-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1272454 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1273460 DEBUG (ScheduledTrigger-4608-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1274466 DEBUG (ScheduledTrigger-4608-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1275423 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== ADD TRIGGERS
   [junit4]   2> 1275433 INFO  (qtp912196633-11743) [n:127.0.0.1:65074_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 1275435 INFO  (qtp1450048634-11797) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1275435 DEBUG (qtp1450048634-11793) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1275437 INFO  (qtp1450048634-11793) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=10
   [junit4]   2> 1275439 DEBUG (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1275440 INFO  (qtp912196633-11745) [n:127.0.0.1:65074_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1275443 INFO  (qtp1450048634-11797) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1275443 DEBUG (qtp1450048634-11795) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1275444 DEBUG (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65074_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1275444 DEBUG (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 1275445 INFO  (qtp1450048634-11795) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 1275446 DEBUG (zkCallback-2644-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65074_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1275447 INFO  (qtp912196633-11748) [n:127.0.0.1:65074_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1275449 DEBUG (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1275450 DEBUG (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65074_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1275451 DEBUG (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65074_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1275451 DEBUG (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 1275452 DEBUG (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65074_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1275453 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1275458 INFO  (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
   [junit4]   2> 1275462 DEBUG (ScheduledTrigger-4608-thread-4) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 2
   [junit4]   2> 1275464 INFO  (qtp912196633-11748) [n:127.0.0.1:65074_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=16
   [junit4]   2> 1275465 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== ADD NODE 1
   [junit4]   2> 1275465 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1275466 DEBUG (ScheduledTrigger-4608-thread-3) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1275468 INFO  (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
   [junit4]   2> 1275468 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1275468 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1275468 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1275469 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@705e2f82{/solr,null,AVAILABLE}
   [junit4]   2> 1275469 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5d8b02e0{HTTP/1.1,[http/1.1]}{127.0.0.1:65103}
   [junit4]   2> 1275469 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.Server Started @1281023ms
   [junit4]   2> 1275469 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65103}
   [junit4]   2> 1275469 ERROR (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1275470 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1275470 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1275470 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1275470 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T08:17:28.070Z
   [junit4]   2> 1275473 DEBUG (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1275474 INFO  (zkConnectionManagerCallback-2667-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1275475 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
   [junit4]   2> 1275475 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1275475 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1275489 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1275491 INFO  (zkConnectionManagerCallback-2671-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1275494 INFO  (zkConnectionManagerCallback-2673-thread-1-processing-n:127.0.0.1:65103_solr) [n:127.0.0.1:65103_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1275507 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1275511 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1275514 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65103_solr
   [junit4]   2> 1275516 INFO  (zkCallback-2662-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1275516 INFO  (zkCallback-2654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1275517 INFO  (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1275517 INFO  (zkCallback-2672-thread-1-processing-n:127.0.0.1:65103_solr) [n:127.0.0.1:65103_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1275672 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65103.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1275706 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65103.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1275706 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65103.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1275709 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [n:127.0.0.1:65103_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/tempDir-001/node4/.
   [junit4]   2> 1276470 DEBUG (ScheduledTrigger-4608-thread-3) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1276471 DEBUG (ScheduledTrigger-4608-thread-1) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 3
   [junit4]   2> 1276490 DEBUG (ScheduledTrigger-4608-thread-4) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
   [junit4]   2> 1276490 DEBUG (ScheduledTrigger-4608-thread-4) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1276490 DEBUG (ScheduledTrigger-4608-thread-4) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:65103_solr at time 1073267902399723
   [junit4]   2> 1277462 INFO  (OverseerCollectionConfigSetProcessor-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1277474 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1277474 DEBUG (ScheduledTrigger-4608-thread-3) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 3
   [junit4]   2> 1277501 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
   [junit4]   2> 1277501 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1277501 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR firing registered processor for nodes: [127.0.0.1:65103_solr] added at times [1073267902399723], now=1073268913324526
   [junit4]   2> 1277501 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3d021a81244ebTck0z0t975uxujq6hs0iuce6ym",
   [junit4]   2>   "source":"node_added_triggerMR",
   [junit4]   2>   "eventTime":1073267902399723,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073267902399723],
   [junit4]   2>     "nodeNames":["127.0.0.1:65103_solr"]}}
   [junit4]   2> 1277502 DEBUG (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_triggerMR, node_added_triggerMR, .auto_add_replicas]
   [junit4]   2> 1277503 INFO  (ScheduledTrigger-4608-thread-2) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1277503 DEBUG (AutoscalingActionExecutor-4609-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3d021a81244ebTck0z0t975uxujq6hs0iuce6ym",
   [junit4]   2>   "source":"node_added_triggerMR",
   [junit4]   2>   "eventTime":1073267902399723,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073267902399723],
   [junit4]   2>     "_enqueue_time_":1073268914098096,
   [junit4]   2>     "nodeNames":["127.0.0.1:65103_solr"]}}
   [junit4]   2> 1277505 DEBUG (AutoscalingActionExecutor-4609-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_triggerMR after 5000ms
   [junit4]   2> 1277505 DEBUG (AutoscalingActionExecutor-4609-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerMR after 5000ms
   [junit4]   2> 1277505 DEBUG (AutoscalingActionExecutor-4609-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 5000ms
   [junit4]   2> 1277506 DEBUG (AutoscalingActionExecutor-4609-thread-1-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=3d021a81244ebTck0z0t975uxujq6hs0iuce6ym
   [junit4]   2> 1280849 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== KILL OVERSEER 2
   [junit4]   2> 1280850 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1dce3a24{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1280851 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=146811364
   [junit4]   2> 1280851 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1280851 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6861523b: rootName = solr_65074, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6a668291
   [junit4]   2> 1280863 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1280863 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6895cbd2: rootName = solr_65074, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@a1c06b6
   [junit4]   2> 1280872 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1280872 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3773f7e8: rootName = solr_65074, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@701e731c
   [junit4]   2> 1280899 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:65074_solr
   [junit4]   2> 1280906 INFO  (zkCallback-2654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1280909 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1280910 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1280910 INFO  (zkCallback-2662-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1280910 INFO  (zkCallback-2644-thread-2-processing-n:127.0.0.1:65074_solr) [n:127.0.0.1:65074_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1280910 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.Overseer Overseer (id=72127931275608070-127.0.0.1:65074_solr-n_0000000001) closing
   [junit4]   2> 1280910 INFO  (OverseerStateUpdate-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:65074_solr
   [junit4]   2> 1280911 INFO  (zkCallback-2672-thread-1-processing-n:127.0.0.1:65103_solr) [n:127.0.0.1:65103_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1280911 DEBUG (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1280912 WARN  (OverseerAutoScalingTriggerThread-72127931275608070-127.0.0.1:65074_solr-n_0000000001) [n:127.0.0.1:65074_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1280917 INFO  (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65083_solr
   [junit4]   2> 1280917 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@18e6eab{/solr,null,UNAVAILABLE}
   [junit4]   2> 1280917 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1280919 INFO  (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.Overseer Overseer (id=72127931275608075-127.0.0.1:65083_solr-n_0000000002) starting
   [junit4]   2> 1280929 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1280930 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1280931 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1280931 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 1280932 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1280932 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion -1
   [junit4]   2> 1280932 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1280932 INFO  (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
   [junit4]   2> 1280933 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:65074_solr
   [junit4]   2> 1280934 INFO  (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
   [junit4]   2> 1280936 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 2
   [junit4]   2> 1280936 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:65074_solr
   [junit4]   2> 1280954 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1280954 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
   [junit4]   2> 1280955 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1280956 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1280956 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:65074_solr
   [junit4]   2> 1281962 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 2
   [junit4]   2> 1281962 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:65074_solr]
   [junit4]   2> 1281967 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3d022b10ca6baTck0z0t975uxujq6hs0iuce6yo",
   [junit4]   2>   "source":"node_lost_triggerMR",
   [junit4]   2>   "eventTime":1073272347993786,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073272347993786],
   [junit4]   2>     "nodeNames":["127.0.0.1:65074_solr"]}}
   [junit4]   2> 1281967 DEBUG (ScheduledTrigger-4619-thread-4) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
   [junit4]   2> 1281967 DEBUG (ScheduledTrigger-4619-thread-4) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1281967 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_triggerMR, node_added_triggerMR, .auto_add_replicas]
   [junit4]   2> 1281971 INFO  (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1281971 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3d022b10ca6baTck0z0t975uxujq6hs0iuce6yo",
   [junit4]   2>   "source":"node_lost_triggerMR",
   [junit4]   2>   "eventTime":1073272347993786,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073272347993786],
   [junit4]   2>     "_enqueue_time_":1073273379827872,
   [junit4]   2>     "nodeNames":["127.0.0.1:65074_solr"]}}
   [junit4]   2> 1281972 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[5063219FE1E594E0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
   [junit4]   2> 1281981 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_triggerMR after 5000ms
   [junit4]   2> 1281981 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerMR after 5000ms
   [junit4]   2> 1281981 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 5000ms
   [junit4]   2> 1281981 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 9 ms for event id=3d022b10ca6baTck0z0t975uxujq6hs0iuce6yo
   [junit4]   2> 1282036 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
   [junit4]   2> 1282037 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 6
   [junit4]   2> 1282038 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1282038 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1282054 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1282055 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1282055 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1282065 INFO  (qtp1450048634-11795) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1282067 INFO  (qtp1206748368-11834) [n:127.0.0.1:65103_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1282067 DEBUG (qtp1206748368-11830) [n:127.0.0.1:65103_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1282078 INFO  (qtp1206748368-11830) [n:127.0.0.1:65103_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=16
   [junit4]   2> 1282078 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1282089 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1282089 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1282089 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1282091 INFO  (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1282092 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1282092 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1282093 DEBUG (ScheduledTrigger-4619-thread-4) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1282093 DEBUG (ScheduledTrigger-4619-thread-4) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1282099 INFO  (qtp1206748368-11836) [n:127.0.0.1:65103_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1282145 INFO  (qtp1450048634-11795) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1282146 DEBUG (qtp1450048634-11798) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1282147 INFO  (qtp1450048634-11798) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=50
   [junit4]   2> 1282147 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 1282148 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1282148 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1282148 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1282148 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1282149 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1282151 INFO  (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1282152 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1282154 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1282154 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1282154 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1282154 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5169165f{/solr,null,AVAILABLE}
   [junit4]   2> 1282155 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7c317478{HTTP/1.1,[http/1.1]}{127.0.0.1:65128}
   [junit4]   2> 1282155 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.e.j.s.Server Started @1287709ms
   [junit4]   2> 1282155 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65128}
   [junit4]   2> 1282155 ERROR (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1282156 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1282156 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1282156 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1282156 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T08:17:34.756Z
   [junit4]   2> 1282158 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1282159 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1282159 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1282159 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1282159 INFO  (zkConnectionManagerCallback-2677-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1282160 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1282172 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65070/solr
   [junit4]   2> 1282174 INFO  (zkConnectionManagerCallback-2681-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1282177 INFO  (zkConnectionManagerCallback-2683-thread-1-processing-n:127.0.0.1:65128_solr) [n:127.0.0.1:65128_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1282184 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1282188 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1282195 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65128_solr
   [junit4]   2> 1282202 INFO  (zkCallback-2654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1282203 INFO  (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1282202 INFO  (zkCallback-2672-thread-1-processing-n:127.0.0.1:65103_solr) [n:127.0.0.1:65103_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1282203 INFO  (zkCallback-2682-thread-1-processing-n:127.0.0.1:65128_solr) [n:127.0.0.1:65128_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1282483 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65128.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1282499 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65128.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1282499 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65128.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4af1cc87
   [junit4]   2> 1282502 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5063219FE1E594E0]) [n:127.0.0.1:65128_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001/tempDir-001/node5/.
   [junit4]   2> 1283097 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1283097 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1283098 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:65128_solr at time 1073274509820122
   [junit4]   2> 1283098 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:65128_solr] added at times [1073274509820122], now=1073274509877334
   [junit4]   2> 1283098 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3d02331e780daTck0z0t975uxujq6hs0iuce6yq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":1073274509820122,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073274509820122],
   [junit4]   2>     "nodeNames":["127.0.0.1:65128_solr"]}}
   [junit4]   2> 1283098 DEBUG (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger2, node_added_trigger1]
   [junit4]   2> 1283101 INFO  (ScheduledTrigger-4619-thread-1) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1283127 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3d02331e780daTck0z0t975uxujq6hs0iuce6yq",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":1073274509820122,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073274509820122],
   [junit4]   2>     "_enqueue_time_":1073274510094280,
   [junit4]   2>     "nodeNames":["127.0.0.1:65128_solr"]}}
   [junit4]   2> 1283127 INFO  (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest action executed from node_added_trigger1
   [junit4]   2> 1283130 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 5000ms
   [junit4]   2> 1283130 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 5000ms
   [junit4]   2> 1283130 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 29 ms for event id=3d02331e780daTck0z0t975uxujq6hs0iuce6yq
   [junit4]   2> 1288137 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:65128_solr at time 1073279550288572
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:65128_solr] added at times [1073279550288572], now=1073279550324434
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3d0245e56f2bcTck0z0t975uxujq6hs0iuce6ys",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":1073279550288572,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073279550288572],
   [junit4]   2>     "nodeNames":["127.0.0.1:65128_solr"]}}
   [junit4]   2> 1288138 DEBUG (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger2, node_added_trigger1]
   [junit4]   2> 1288139 INFO  (ScheduledTrigger-4619-thread-2) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1288139 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3d0245e56f2bcTck0z0t975uxujq6hs0iuce6ys",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":1073279550288572,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1073279550288572],
   [junit4]   2>     "_enqueue_time_":1073279550479298,
   [junit4]   2>     "nodeNames":["127.0.0.1:65128_solr"]}}
   [junit4]   2> 1288140 INFO  (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest last action at 1517559455727000000 current time = 1517559460739000000
   [junit4]   2> real diff: 5012000000
   [junit4]   2>  min diff: 4998000000
   [junit4]   2> 1288140 INFO  (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest action executed from node_added_trigger2
   [junit4]   2> 1288144 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 5000ms
   [junit4]   2> 1288144 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 5000ms
   [junit4]   2> 1288144 DEBUG (AutoscalingActionExecutor-4620-thread-1-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 4 ms for event id=3d0245e56f2bcTck0z0t975uxujq6hs0iuce6ys
   [junit4]   2> 1288145 INFO  (qtp1450048634-11798) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1288146 INFO  (qtp1206748368-11836) [n:127.0.0.1:65103_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1288148 INFO  (qtp157841741-11870) [n:127.0.0.1:65128_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1288148 DEBUG (qtp157841741-11866) [n:127.0.0.1:65128_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1288149 INFO  (qtp157841741-11866) [n:127.0.0.1:65128_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 1288149 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 1288150 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65128_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1288150 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1288150 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65128_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1288151 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65128_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1288151 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1288151 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1288153 INFO  (qtp1450048634-11798) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1288154 INFO  (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1288155 DEBUG (OverseerAutoScalingTriggerThread-72127931275608075-127.0.0.1:65083_solr-n_0000000002) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1288156 DEBUG (ScheduledTrigger-4619-thread-3) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 3
   [junit4]   2> 1288156 INFO  (qtp1206748368-11832) [n:127.0.0.1:65103_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1288158 INFO  (qtp157841741-11870) [n:127.0.0.1:65128_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1288158 DEBUG (qtp1450048634-11799) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1288159 INFO  (qtp1450048634-11799) [n:127.0.0.1:65083_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=8
   [junit4]   2> 1288159 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 1288160 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65128_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1288160 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1288161 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65128_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1288161 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65103_solr, 127.0.0.1:65128_solr, 127.0.0.1:65083_solr]
   [junit4]   2> 1288161 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1288162 DEBUG (zkCallback-2662-thread-2-processing-n:127.0.0.1:65083_solr) [n:127.0.0.1:65083_solr    ] o.a.s.c.a.NodeLostTrigger Initial

[...truncated too long message...]

-processing-n:127.0.0.1:65208_solr) [n:127.0.0.1:65208_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65208_solr
   [junit4]   2> 1480043 INFO  (jetty-closer-2627-thread-8) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@391f7411{/solr,null,UNAVAILABLE}
   [junit4]   2> 1480047 INFO  (jetty-closer-2627-thread-8) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1480056 INFO  (coreCloseExecutor-4823-thread-1) [n:127.0.0.1:65208_solr c:collection1 s:shard1 r:core_node12 x:collection1_shard1_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1172107765
   [junit4]   2> 1480058 INFO  (coreCloseExecutor-4820-thread-1) [n:127.0.0.1:65323_solr c:collection1 s:shard1 r:core_node14 x:collection1_shard1_replica_n13] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=286240222
   [junit4]   2> 1480060 INFO  (coreCloseExecutor-4824-thread-1) [n:127.0.0.1:65315_solr c:collection1 s:shard1 r:core_node8 x:collection1_shard1_replica_n7] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1942856349
   [junit4]   2> 1480060 INFO  (coreCloseExecutor-4818-thread-1) [n:127.0.0.1:65218_solr c:collection1 s:shard1 r:core_node10 x:collection1_shard1_replica_n9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=94716771
   [junit4]   2> 1480082 INFO  (jetty-closer-2627-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@33677d54{/solr,null,UNAVAILABLE}
   [junit4]   2> 1480083 INFO  (jetty-closer-2627-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1480083 INFO  (zkCallback-2726-thread-2-processing-n:127.0.0.1:65218_solr) [n:127.0.0.1:65218_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65218_solr
   [junit4]   2> 1481603 INFO  (jetty-closer-2627-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@196afdae{/solr,null,UNAVAILABLE}
   [junit4]   2> 1481603 INFO  (jetty-closer-2627-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1481604 INFO  (jetty-closer-2627-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@23b9f366{/solr,null,UNAVAILABLE}
   [junit4]   2> 1481605 INFO  (jetty-closer-2627-thread-7) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1481605 INFO  (jetty-closer-2627-thread-9) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@323e6151{/solr,null,UNAVAILABLE}
   [junit4]   2> 1481605 INFO  (jetty-closer-2627-thread-9) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1481627 ERROR (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1481628 INFO  (SUITE-TriggerIntegrationTest-seed#[5063219FE1E594E0]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:65070 65070
   [junit4]   2> 1481640 INFO  (Thread-2613) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:65070 65070
   [junit4]   2> 1481943 WARN  (Thread-2613) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 44 /solr/collections/collection1/terms/shard1
   [junit4]   2> 23 /solr/clusterprops.json
   [junit4]   2> 23 /solr/aliases.json
   [junit4]   2> 22 /solr/security.json
   [junit4]   2> 12 /solr/configs/conf
   [junit4]   2> 9 /solr/collections/testMetricTrigger/terms/shard1
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard2
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 160 /solr/collections/collection1/state.json
   [junit4]   2> 49 /solr/collections/testMetricTrigger/state.json
   [junit4]   2> 23 /solr/clusterstate.json
   [junit4]   2> 6 /solr/autoscaling.json
   [junit4]   2> 3 /solr/overseer_elect/election/72127931275608115-127.0.0.1:65323_solr-n_0000000015
   [junit4]   2> 3 /solr/overseer_elect/election/72127931275608094-127.0.0.1:65218_solr-n_0000000008
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72127931275608112-core_node3-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72127931275608100-127.0.0.1:65254_solr-n_0000000010
   [junit4]   2> 2 /solr/overseer_elect/election/72127931275608078-127.0.0.1:65103_solr-n_0000000003
   [junit4]   2> 2 /solr/overseer_elect/election/72127931275608109-127.0.0.1:65315_solr-n_0000000013
   [junit4]   2> 2 /solr/overseer_elect/election/72127931275608118-127.0.0.1:65422_solr-n_0000000016
   [junit4]   2> 2 /solr/overseer_elect/election/72127931275608127-127.0.0.1:65507_solr-n_0000000019
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72127931275608094-core_node10-n_0000000004
   [junit4]   2> 2 /solr/overseer_elect/election/72127931275608124-127.0.0.1:65493_solr-n_0000000018
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 23 /solr/live_nodes
   [junit4]   2> 23 /solr/collections
   [junit4]   2> 6 /solr/overseer/queue-work
   [junit4]   2> 6 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 6 /solr/overseer/queue
   [junit4]   2> 6 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/autoscaling/events/node_lost_triggerMR
   [junit4]   2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5063219FE1E594E0-001
   [junit4]   2> Feb 02, 2018 8:20:54 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3b347b99), locale=sr-Latn-BA, timezone=Asia/Vladivostok
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=129083992,total=306716672
   [junit4]   2> NOTE: All tests run in this JVM: [TestSubQueryTransformerDistrib, CdcrUpdateLogTest, SimpleCollectionCreateDeleteTest, TestReversedWildcardFilterFactory, TestSQLHandlerNonCloud, TestDynamicFieldCollectionResource, TestLuceneMatchVersion, BooleanFieldTest, TestFieldSortValues, TestCollationFieldDocValues, JSONWriterTest, HdfsNNFailoverTest, CurrencyFieldTypeTest, TestLegacyFieldReuse, HdfsLockFactoryTest, TestBlobHandler, TestMaxScoreQueryParser, ClusterStateTest, TestReplicationHandlerBackup, BadComponentTest, HdfsAutoAddReplicasIntegrationTest, TestPullReplicaErrorHandling, TestSchemaNameResource, ShardRoutingTest, DistributedFacetSimpleRefinementLongTailTest, TestStressUserVersions, TestBulkSchemaAPI, AliasIntegrationTest, DeleteReplicaTest, PKIAuthenticationIntegrationTest, TestLargeCluster, NotRequiredUniqueKeyTest, SpellCheckComponentTest, TlogReplayBufferedWhileIndexingTest, IndexBasedSpellCheckerTest, DistanceUnitsTest, TestOverriddenPrefixQueryForCustomFieldType, DistributedFacetPivotLargeTest, SolrCmdDistributorTest, ReturnFieldsTest, NodeLostTriggerTest, UpdateParamsTest, TestFieldCache, OverseerRolesTest, TestCollectionsAPIViaSolrCloudCluster, TestCloudDeleteByQuery, AnalyticsQueryTest, ConcurrentCreateRoutedAliasTest, SubstringBytesRefFilterTest, DistribDocExpirationUpdateProcessorTest, TestSolrIndexConfig, TestRestoreCore, TestStressVersions, DocValuesMissingTest, TestBlendedInfixSuggestions, ReplicaListTransformerTest, TestIntervalFaceting, TestDocSet, ResponseHeaderTest, TestLRUCache, TestExportWriter, ReplaceNodeNoTargetTest, TestStressLucene, ExitableDirectoryReaderTest, TestDistributedSearch, TestJettySolrRunner, TestConfig, TestFastWriter, TestPrepRecovery, DistributedIntervalFacetingTest, SpatialHeatmapFacetsTest, CoreSorterTest, TestIndexingPerformance, ZkStateWriterTest, SliceStateTest, TestDeleteCollectionOnDownNodes, CollectionStateFormat2Test, IndexSchemaTest, SyncSliceTest, ClassificationUpdateProcessorFactoryTest, TestUpdate, TestClusterStateMutator, ResponseLogComponentTest, ConcurrentDeleteAndCreateCollectionTest, FieldAnalysisRequestHandlerTest, TestDistributedMissingSort, HdfsRecoveryZkTest, HealthCheckHandlerTest, URLClassifyProcessorTest, TimeZoneUtilsTest, DeleteStatusTest, TestRandomCollapseQParserPlugin, TestWriterPerf, SolrShardReporterTest, MoveReplicaHDFSTest, TestSearchPerf, ScriptEngineTest, TestReloadDeadlock, SolrCoreMetricManagerTest, TestConfigSetsAPI, TestNRTOpen, TestPivotHelperCode, QueryParsingTest, CdcrVersionReplicationTest, CreateRoutedAliasTest, OverseerTaskQueueTest, TestGraphMLResponseWriter, VersionInfoTest, TestXmlQParser, TestDistribIDF, TestCollapseQParserPlugin, InfoHandlerTest, HdfsChaosMonkeyNothingIsSafeTest, SuggesterTest, TestSolrQueryParser, TestMaxTokenLenTokenizer, BadIndexSchemaTest, TestGraphTermsQParserPlugin, ExternalFileFieldSortTest, DocumentAnalysisRequestHandlerTest, AtomicUpdateProcessorFactoryTest, SchemaApiFailureTest, PrimUtilsTest, PeerSyncWithIndexFingerprintCachingTest, TestSha256AuthenticationProvider, DisMaxRequestHandlerTest, TestSchemaManager, TestHashPartitioner, TestLeaderElectionWithEmptyReplica, TestScoreJoinQPScore, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, SuggesterWFSTTest, TestRawTransformer, DocValuesNotIndexedTest, TestFuzzyAnalyzedSuggestions, CleanupOldIndexTest, TriggerIntegrationTest]
   [junit4] Completed [273/775 (1!)] on J1 in 212.98s, 14 tests, 1 failure <<< FAILURES!

[...truncated 49774 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4421 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4421/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly

Error Message:
Unexpected number of elements in the group for intGSF: 5

Stack Trace:
java.lang.AssertionError: Unexpected number of elements in the group for intGSF: 5
        at __randomizedtesting.SeedInfo.seed([9237FDC76FE72617:98C939F22BF1449]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly(DocValuesNotIndexedTest.java:377)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testSetProperties

Error Message:
Expected 8 triggers but found: [x0, x1, x2, x3, x4, x5, x6] expected:<8> but was:<7>

Stack Trace:
java.lang.AssertionError: Expected 8 triggers but found: [x0, x1, x2, x3, x4, x5, x6] expected:<8> but was:<7>
        at __randomizedtesting.SeedInfo.seed([9237FDC76FE72617:F9532A8EDCCAB393]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testSetProperties(TriggerIntegrationTest.java:1334)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 1794 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/core/test/temp/junit4-J1-20180202_184718_5237562541064479045137.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/core/test/temp/junit4-J0-20180202_184718_5238045851342677213786.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 288 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/test-framework/test/temp/junit4-J1-20180202_185526_6326145791605238480206.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/test-framework/test/temp/junit4-J0-20180202_185526_6322322593291621615262.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 1057 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/common/test/temp/junit4-J0-20180202_185647_9362360473414184877084.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/common/test/temp/junit4-J1-20180202_185647_9369964986841887378677.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 224 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/icu/test/temp/junit4-J1-20180202_185902_1941025410494399663353.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 5 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/icu/test/temp/junit4-J0-20180202_185902_1944748508516940684356.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 237 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/kuromoji/test/temp/junit4-J0-20180202_185911_7057072383294968886217.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 15 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/kuromoji/test/temp/junit4-J1-20180202_185911_70515334983663389944683.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 154 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/morfologik/test/temp/junit4-J1-20180202_185940_50010510335019752401540.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/morfologik/test/temp/junit4-J0-20180202_185940_5007753630894174699490.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 161 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/opennlp/test/temp/junit4-J0-20180202_185944_7857300885573189873260.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/opennlp/test/temp/junit4-J1-20180202_185944_78511218087677536823936.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 171 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/phonetic/test/temp/junit4-J0-20180202_185948_14818377908013884293014.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/phonetic/test/temp/junit4-J1-20180202_185948_14818060513418888203825.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 151 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/smartcn/test/temp/junit4-J1-20180202_185958_4871548607978022720440.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/smartcn/test/temp/junit4-J0-20180202_185958_4879181519765132399609.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 159 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/stempel/test/temp/junit4-J1-20180202_190011_23013420545059200379938.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/stempel/test/temp/junit4-J0-20180202_190011_2301770601309413655700.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 167 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/uima/test/temp/junit4-J0-20180202_190014_4844006497307067746277.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 152 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/backward-codecs/test/temp/junit4-J0-20180202_190027_9055792777138194186394.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 19 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/backward-codecs/test/temp/junit4-J1-20180202_190027_9066337670359869735260.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 1304 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/benchmark/test/temp/junit4-J1-20180202_190052_2762987242691733904432.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/benchmark/test/temp/junit4-J0-20180202_190052_27612631549405195866636.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 231 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/classification/test/temp/junit4-J1-20180202_190102_5748997192163470213830.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/classification/test/temp/junit4-J0-20180202_190102_574250380675957827195.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 259 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/codecs/test/temp/junit4-J1-20180202_190119_54615611257877663588312.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/codecs/test/temp/junit4-J0-20180202_190119_54612043988147020569594.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 212 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/demo/test/temp/junit4-J0-20180202_190325_17411016713370879780425.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/demo/test/temp/junit4-J1-20180202_190325_1742289485491957999985.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 170 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/expressions/test/temp/junit4-J1-20180202_190330_16110546054803512023719.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/expressions/test/temp/junit4-J0-20180202_190330_16114923671126084400385.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 226 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/facet/test/temp/junit4-J1-20180202_190336_8228827004567917075076.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/facet/test/temp/junit4-J0-20180202_190336_8219476093232365243527.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 179 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/grouping/test/temp/junit4-J0-20180202_190407_6718369169243382746962.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/grouping/test/temp/junit4-J1-20180202_190407_67110091498377879119080.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 243 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/highlighter/test/temp/junit4-J1-20180202_190419_23817630873386108241732.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/highlighter/test/temp/junit4-J0-20180202_190419_23613770255791054591907.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 163 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/join/test/temp/junit4-J0-20180202_190447_0733032527006561930584.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/join/test/temp/junit4-J1-20180202_190447_073293342926886048301.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 147 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/memory/test/temp/junit4-J0-20180202_190500_27110504054547554350360.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/memory/test/temp/junit4-J1-20180202_190500_27116544186600879826662.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 179 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/misc/test/temp/junit4-J0-20180202_190506_84314673883807687849660.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/misc/test/temp/junit4-J1-20180202_190506_84317057795535348799265.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 298 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queries/test/temp/junit4-J1-20180202_190524_00713277495815566821760.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queries/test/temp/junit4-J0-20180202_190524_00710731853757211124271.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 227 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queryparser/test/temp/junit4-J1-20180202_190533_75118083048585074760253.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queryparser/test/temp/junit4-J0-20180202_190533_7517758957728414383614.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 187 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/replicator/test/temp/junit4-J0-20180202_190542_2768985376128022441359.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 23 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/replicator/test/temp/junit4-J1-20180202_190542_2763441488586290967357.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 223 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/sandbox/test/temp/junit4-J0-20180202_190555_38316159309829438670130.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/sandbox/test/temp/junit4-J1-20180202_190555_38310096778558262030073.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 317 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial-extras/test/temp/junit4-J1-20180202_190627_6606786450381792660792.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial-extras/test/temp/junit4-J0-20180202_190627_6606781952407559445033.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 146 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial/test/temp/junit4-J0-20180202_190644_5408368558221797182323.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 180 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial3d/test/temp/junit4-J1-20180202_190646_81110095342423378641773.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 5 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial3d/test/temp/junit4-J0-20180202_190646_81116672813401133002489.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 253 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/suggest/test/temp/junit4-J1-20180202_190704_58214097640072204216262.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/suggest/test/temp/junit4-J0-20180202_190704_5825760323338375957429.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 2824 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> 1609396 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_9237FDC76FE72617-001/init-core-data-001
   [junit4]   2> 1609402 WARN  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1609402 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1609403 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1609404 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_9237FDC76FE72617-001/tempDir-001
   [junit4]   2> 1609404 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1609404 INFO  (Thread-3077) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1609404 INFO  (Thread-3077) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1609407 ERROR (Thread-3077) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1609510 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:59232
   [junit4]   2> 1609520 INFO  (zkConnectionManagerCallback-3290-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609527 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1609528 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1609531 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1609531 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1609531 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1609531 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1609532 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1609532 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1609532 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2bf52a40{/solr,null,AVAILABLE}
   [junit4]   2> 1609532 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@523c2d95{/solr,null,AVAILABLE}
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@70148e6c{SSL,[ssl, http/1.1]}{127.0.0.1:59234}
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.e.j.s.Server Started @1611265ms
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59234}
   [junit4]   2> 1609533 ERROR (jetty-launcher-3287-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1609533 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T19:35:00.224939Z
   [junit4]   2> 1609535 INFO  (zkConnectionManagerCallback-3292-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609536 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1609543 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4e4ac77{SSL,[ssl, http/1.1]}{127.0.0.1:59235}
   [junit4]   2> 1609543 INFO  (jetty-launcher-3287-thread-1) [    ] o.e.j.s.Server Started @1611275ms
   [junit4]   2> 1609543 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59235}
   [junit4]   2> 1609543 ERROR (jetty-launcher-3287-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1609544 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1609544 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1609544 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1609544 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T19:35:00.235250Z
   [junit4]   2> 1609546 INFO  (zkConnectionManagerCallback-3294-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609547 INFO  (jetty-launcher-3287-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59232/solr
   [junit4]   2> 1609548 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1609550 INFO  (zkConnectionManagerCallback-3298-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609568 INFO  (zkConnectionManagerCallback-3300-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609582 INFO  (jetty-launcher-3287-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59232/solr
   [junit4]   2> 1609585 INFO  (zkConnectionManagerCallback-3304-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609589 INFO  (zkConnectionManagerCallback-3306-thread-1-processing-n:127.0.0.1:59235_solr) [n:127.0.0.1:59235_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1609974 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1609977 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59234_solr
   [junit4]   2> 1609978 INFO  (jetty-launcher-3287-thread-1) [n:127.0.0.1:59235_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1609979 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.Overseer Overseer (id=72130595827875844-127.0.0.1:59234_solr-n_0000000000) starting
   [junit4]   2> 1609981 INFO  (jetty-launcher-3287-thread-1) [n:127.0.0.1:59235_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59235_solr
   [junit4]   2> 1609985 INFO  (zkCallback-3305-thread-1-processing-n:127.0.0.1:59235_solr) [n:127.0.0.1:59235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1609987 INFO  (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1610015 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1610019 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1610021 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59235_solr]
   [junit4]   2> 1610021 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1610021 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1610022 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59234_solr
   [junit4]   2> 1610024 INFO  (zkCallback-3305-thread-1-processing-n:127.0.0.1:59235_solr) [n:127.0.0.1:59235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1610026 INFO  (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1610029 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1610031 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1610031 DEBUG (ScheduledTrigger-6720-thread-1) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1610054 INFO  (jetty-launcher-3287-thread-1) [n:127.0.0.1:59235_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59235.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610124 INFO  (jetty-launcher-3287-thread-1) [n:127.0.0.1:59235_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59235.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610124 INFO  (jetty-launcher-3287-thread-1) [n:127.0.0.1:59235_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59235.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610130 INFO  (jetty-launcher-3287-thread-1) [n:127.0.0.1:59235_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_9237FDC76FE72617-001/tempDir-001/node1/.
   [junit4]   2> 1610221 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59234.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610239 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59234.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610240 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59234.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610241 INFO  (jetty-launcher-3287-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_9237FDC76FE72617-001/tempDir-001/node2/.
   [junit4]   2> 1610270 INFO  (zkConnectionManagerCallback-3312-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1610272 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10042650dc20007, likely client has closed socket
   [junit4]   2> 1610274 INFO  (zkConnectionManagerCallback-3316-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1610276 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1610278 INFO  (SUITE-TriggerIntegrationTest-seed#[9237FDC76FE72617]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59232/solr ready
   [junit4]   2> 1610386 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testListeners
   [junit4]   2> 1610387 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 1610387 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1610388 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1610406 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1610406 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1610414 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1610543 INFO  (qtp2110287456-15481) [n:127.0.0.1:59234_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1610548 INFO  (qtp1762766674-15482) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1610549 DEBUG (qtp1762766674-15480) [n:127.0.0.1:59235_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1610549 INFO  (qtp1762766674-15480) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=51
   [junit4]   2> 1610550 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1610566 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59234_solr, 127.0.0.1:59235_solr]
   [junit4]   2> 1610566 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction}], enabled=true}
   [junit4]   2> 1610567 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1610569 INFO  (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1610571 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1610571 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1610572 DEBUG (ScheduledTrigger-6720-thread-1) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1610572 DEBUG (ScheduledTrigger-6720-thread-1) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1610592 INFO  (qtp2110287456-15485) [n:127.0.0.1:59234_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1610594 INFO  (qtp1762766674-15480) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1610595 DEBUG (qtp1762766674-15477) [n:127.0.0.1:59235_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1610595 INFO  (qtp1762766674-15477) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=14
   [junit4]   2> 1610596 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1610597 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59234_solr, 127.0.0.1:59235_solr]
   [junit4]   2> 1610597 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction}], enabled=true}
   [junit4]   2> 1610597 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1610598 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1610598 INFO  (qtp2110287456-15481) [n:127.0.0.1:59234_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1610598 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1610599 INFO  (qtp1762766674-15477) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1610599 DEBUG (qtp1762766674-15482) [n:127.0.0.1:59235_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1610600 INFO  (qtp1762766674-15482) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1610600 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1610601 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59234_solr, 127.0.0.1:59235_solr]
   [junit4]   2> 1610601 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction}], enabled=true}
   [junit4]   2> 1610601 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1610602 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1610603 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1610606 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1610607 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1610607 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1610607 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1610608 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c0fd41d{/solr,null,AVAILABLE}
   [junit4]   2> 1610608 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f73906a{SSL,[ssl, http/1.1]}{127.0.0.1:59250}
   [junit4]   2> 1610609 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.Server Started @1612341ms
   [junit4]   2> 1610609 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59250}
   [junit4]   2> 1610609 ERROR (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1610609 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1610609 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1610609 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1610609 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T19:35:01.300918Z
   [junit4]   2> 1610638 INFO  (zkConnectionManagerCallback-3318-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1610649 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1610664 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59232/solr
   [junit4]   2> 1610667 INFO  (zkConnectionManagerCallback-3322-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1610683 INFO  (zkConnectionManagerCallback-3324-thread-1-processing-n:127.0.0.1:59250_solr) [n:127.0.0.1:59250_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1610700 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1610703 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1610707 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59250_solr
   [junit4]   2> 1610716 INFO  (zkCallback-3305-thread-1-processing-n:127.0.0.1:59235_solr) [n:127.0.0.1:59235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1610716 INFO  (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1610717 INFO  (zkCallback-3315-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1610717 INFO  (zkCallback-3323-thread-1-processing-n:127.0.0.1:59250_solr) [n:127.0.0.1:59250_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1610799 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59250.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610816 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59250.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610816 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59250.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1610817 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59250_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_9237FDC76FE72617-001/tempDir-001/node3/.
   [junit4]   2> 1611614 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1611614 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1611614 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59250_solr at time 1113921122664066
   [junit4]   2> 1612625 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1612626 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1613632 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1613632 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1613632 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL firing registered processor for nodes: [127.0.0.1:59250_solr] added at times [1113921122664066], now=1113923140117830
   [junit4]   2> 1613632 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3f51af8d2fe82T1659cdmqae77emg9xscy0qly7",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1113921122664066,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1113921122664066],
   [junit4]   2>     "nodeNames":["127.0.0.1:59250_solr"]}}
   [junit4]   2> 1613632 DEBUG (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerL]
   [junit4]   2> 1613633 INFO  (ScheduledTrigger-6720-thread-2) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1613634 DEBUG (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3f51af8d2fe82T1659cdmqae77emg9xscy0qly7",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1113921122664066,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1113921122664066],
   [junit4]   2>     "_enqueue_time_":1113923140757624,
   [junit4]   2>     "nodeNames":["127.0.0.1:59250_solr"]}}
   [junit4]   2> 1613636 DEBUG (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerL after 5000ms
   [junit4]   2> 1613637 DEBUG (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=3f51af8d2fe82T1659cdmqae77emg9xscy0qly7
   [junit4]   2> 1615642 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1615644 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1615644 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1615644 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1615644 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@24a70315{/solr,null,AVAILABLE}
   [junit4]   2> 1615645 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6a04e391{SSL,[ssl, http/1.1]}{127.0.0.1:59273}
   [junit4]   2> 1615645 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.e.j.s.Server Started @1617377ms
   [junit4]   2> 1615645 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59273}
   [junit4]   2> 1615645 ERROR (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1615670 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1615670 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1615670 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1615670 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T19:35:06.361926Z
   [junit4]   2> 1615673 INFO  (zkConnectionManagerCallback-3328-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1615675 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1615701 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59232/solr
   [junit4]   2> 1615704 INFO  (zkConnectionManagerCallback-3332-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1615724 INFO  (zkConnectionManagerCallback-3334-thread-1-processing-n:127.0.0.1:59273_solr) [n:127.0.0.1:59273_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1615736 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1615740 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1615744 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59273_solr
   [junit4]   2> 1615747 INFO  (zkCallback-3305-thread-1-processing-n:127.0.0.1:59235_solr) [n:127.0.0.1:59235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1615747 INFO  (zkCallback-3315-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1615747 INFO  (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1615747 INFO  (zkCallback-3323-thread-1-processing-n:127.0.0.1:59250_solr) [n:127.0.0.1:59250_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1615748 INFO  (zkCallback-3333-thread-1-processing-n:127.0.0.1:59273_solr) [n:127.0.0.1:59273_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1615824 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59273.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1615843 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59273.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1615843 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59273.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@592219ed
   [junit4]   2> 1615845 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [n:127.0.0.1:59273_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_9237FDC76FE72617-001/tempDir-001/node4/.
   [junit4]   2> 1618647 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1618647 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1618647 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59273_solr at time 1113928155333791
   [junit4]   2> 1619649 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1619649 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1620655 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 1620655 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1620655 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL firing registered processor for nodes: [127.0.0.1:59273_solr] added at times [1113928155333791], now=1113930163186969
   [junit4]   2> 1620655 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"3f51c9c01049fT1659cdmqae77emg9xscy0qly9",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1113928155333791,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1113928155333791],
   [junit4]   2>     "nodeNames":["127.0.0.1:59273_solr"]}}
   [junit4]   2> 1620655 DEBUG (ScheduledTrigger-6720-thread-4) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerL]
   [junit4]   2> 1620656 DEBUG (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"3f51c9c01049fT1659cdmqae77emg9xscy0qly9",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1113928155333791,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1113928155333791],
   [junit4]   2>     "_enqueue_time_":1113930163397727,
   [junit4]   2>     "nodeNames":["127.0.0.1:59273_solr"]}}
   [junit4]   2> 1620656 WARN  (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: test1 for trigger event: {
   [junit4]   2>   "id":"3f51c9c01049fT1659cdmqae77emg9xscy0qly9",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1113928155333791,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1113928155333791],
   [junit4]   2>     "_enqueue_time_":1113930163397727,
   [junit4]   2>     "nodeNames":["127.0.0.1:59273_solr"]}}
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:302)
   [junit4]   2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
   [junit4]   2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
   [junit4]   2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4]   2> at java.base/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Caused by: java.lang.RuntimeException: failure
   [junit4]   2> at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction.process(TriggerIntegrationTest.java:989)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$3(ScheduledTriggers.java:299)
   [junit4]   2> ... 6 more
   [junit4]   2> 1620657 DEBUG (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerL after 5000ms
   [junit4]   2> 1620657 DEBUG (AutoscalingActionExecutor-6721-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 0 ms for event id=3f51c9c01049fT1659cdmqae77emg9xscy0qly9
   [junit4]   2> 1622686 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[9237FDC76FE72617]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testListeners
   [junit4]   2> 1622793 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
   [junit4]   2> 1622794 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1622794 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1622796 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1622796 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1622797 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1622797 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 6
   [junit4]   2> 1622874 INFO  (qtp2110287456-15487) [n:127.0.0.1:59234_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1622895 INFO  (qtp1762766674-15480) [n:127.0.0.1:59235_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1622928 INFO  (qtp1115441777-15542) [n:127.0.0.1:59250_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1622944 INFO  (qtp959133884-15568) [n:127.0.0.1:59273_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1622945 DEBUG (qtp2110287456-15485) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1622945 INFO  (qtp2110287456-15485) [n:127.0.0.1:59234_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=76
   [junit4]   2> 1622946 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1622946 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59234_solr, 127.0.0.1:59250_solr, 127.0.0.1:59273_solr, 127.0.0.1:59235_solr]
   [junit4]   2> 1622947 DEBUG (zkCallback-3299-thread-1-processing-n:127.0.0.1:59234_solr) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEFRS instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1622947 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1622949 INFO  (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1622950 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1622950 INFO  (qtp2110287456-15486) [n:127.0.0.1:59234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1622950 DEBUG (OverseerAutoScalingTriggerThread-72130595827875844-127.0.0.1:59234_solr-n_0000000000) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1622951 DEBUG (ScheduledTrigger-6720-thread-3) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
   [junit4]   2> 1622951 DEBUG (ScheduledTrigger-6720-thread-3) [n:127.0.0.1:59234_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1622964 INFO  (qtp2110287456-15486) [n:127.0.0.1:59234_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=14
   [junit4]   2> 1622968 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T14:27:37-07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1622970 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1622970 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1622970 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1622971 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c87d176{/solr,null,AVAILABLE}
   [junit4]   2> 1622971 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@45e2fa62{SSL,[ssl, http/1.1]}{127.0.0.1:59311}
   [junit4]   2> 1622971 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.e.j.s.Server Started @1624703ms
   [junit4]   2> 1622971 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59311}
   [junit4]   2> 1622972 ERROR (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1622972 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1622972 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1622972 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1622972 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-02T19:35:13.663592Z
   [junit4]   2> 1622974 INFO  (zkConnectionManagerCallback-3338-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1622975 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1622998 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59232/solr
   [junit4]   2> 1623000 INFO  (zkConnectionManagerCallback-3342-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1623008 INFO  (zkConnectionManagerCallback-3344-thread-1-processing-n:127.0.0.1:59311_solr) [n:127.0.0.1:59311_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1623039 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [n:127.0.0.1:59311_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1623042 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [n:127.0.0.1:59311_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1623061 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[9237FDC76FE72617]) [n:127.0.0.1:59311_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59311_solr
   [junit4]   2> 1623068 INFO  (zkCallback-3305-thread-1-processing-n:127.0.0.1:59235_solr) [n:127.0.0.1:59235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1623068 INFO  (zkCallback-3315-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1623069 INFO  (zkCallback-3323-thread-1-processing-n:127.0.0.1:59250_solr) [n:127.0.0.1:59250_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1623069 INFO  (zkCallback-3333-thread-1-processi

[...truncated too long message...]

solr/build/solr-solrj/test/temp/junit4-J1-20180202_201703_2382303923941531648543.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/temp/junit4-J0-20180202_201703_2389234263680945187196.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 1463 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-analysis-extras/test/temp/junit4-J0-20180202_202129_07810453585365295708260.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-analysis-extras/test/temp/junit4-J1-20180202_202129_07910574944193732358010.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 785 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-analytics/test/temp/junit4-J1-20180202_202146_34913973017221661519745.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-analytics/test/temp/junit4-J0-20180202_202146_3499676039058679680875.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 553 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-clustering/test/temp/junit4-J1-20180202_202247_9272899707773266746415.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-clustering/test/temp/junit4-J0-20180202_202247_92713255718766123086495.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 1258 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-dataimporthandler-extras/test/temp/junit4-J1-20180202_202308_8834373338169078091092.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-dataimporthandler-extras/test/temp/junit4-J0-20180202_202308_88311171403659777349686.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 646 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J1-20180202_202319_89613047497843924539569.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20180202_202319_8965397770893705596838.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 555 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-cell/test/temp/junit4-J1-20180202_202347_646418573421025928692.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 6 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-cell/test/temp/junit4-J0-20180202_202347_64614511824088407112834.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 561 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-langid/test/temp/junit4-J1-20180202_202404_0266612230895882791570.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-langid/test/temp/junit4-J0-20180202_202404_02610862158379568371908.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 655 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-ltr/test/temp/junit4-J1-20180202_202414_405140464230543010066.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-ltr/test/temp/junit4-J0-20180202_202414_4058014906938103777435.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 595 lines...]
   [junit4] JVM J1: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-uima/test/temp/junit4-J1-20180202_202447_8771263094849484314332.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J1: EOF ----

[...truncated 3 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-uima/test/temp/junit4-J0-20180202_202447_8772443691395029435607.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 548 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-velocity/test/temp/junit4-J0-20180202_202504_6354631939422804367852.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   [junit4] <<< JVM J0: EOF ----

[...truncated 32285 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4422 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4422/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseParallelGC

4 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
        at __randomizedtesting.SeedInfo.seed([D1A9C2BFDD39C093:59FDFD6573C5AD6B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:915)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testSelectedCollections

Error Message:
The operations computed by ComputePlanAction should not be nullSolrClientNodeStateProvider.DEBUG{AFTER_ACTION=[compute_plan, null], BEFORE_ACTION=[compute_plan, null]}

Stack Trace:
java.lang.AssertionError: The operations computed by ComputePlanAction should not be nullSolrClientNodeStateProvider.DEBUG{AFTER_ACTION=[compute_plan, null], BEFORE_ACTION=[compute_plan, null]}
        at __randomizedtesting.SeedInfo.seed([D1A9C2BFDD39C093:EB072766E35D19FD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testSelectedCollections(ComputePlanActionTest.java:469)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.sql.JdbcTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.sql.JdbcTest:     1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]         at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.base@9/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.sql.JdbcTest:
   1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
        at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.base@9/java.lang.Thread.run(Thread.java:844)
        at __randomizedtesting.SeedInfo.seed([DE1A845813A31629]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.sql.JdbcTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]         at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.base@9/java.lang.Thread.run(Thread.java:844)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
        at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.base@9/java.lang.Thread.run(Thread.java:844)
        at __randomizedtesting.SeedInfo.seed([DE1A845813A31629]:0)




Build Log:
[...truncated 12480 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.ComputePlanActionTest
   [junit4]   2> 768527 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ComputePlanActionTest_D1A9C2BFDD39C093-001/init-core-data-001
   [junit4]   2> 768528 WARN  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=23 numCloses=23
   [junit4]   2> 768529 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 768530 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 768530 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ComputePlanActionTest_D1A9C2BFDD39C093-001/tempDir-001
   [junit4]   2> 768530 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 768531 INFO  (Thread-1564) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 768531 INFO  (Thread-1564) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 768534 ERROR (Thread-1564) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 768631 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:55775
   [junit4]   2> 768642 INFO  (zkConnectionManagerCallback-1913-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768646 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 768647 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 768647 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 768647 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 768647 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4eb38c29{/solr,null,AVAILABLE}
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6a451563{HTTP/1.1,[http/1.1]}{127.0.0.1:55777}
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.e.j.s.Server Started @770246ms
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55777}
   [junit4]   2> 768649 ERROR (jetty-launcher-1910-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 768649 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T06:16:33.205798Z
   [junit4]   2> 768651 INFO  (zkConnectionManagerCallback-1915-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768652 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 768657 INFO  (jetty-launcher-1910-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55775/solr
   [junit4]   2> 768658 INFO  (zkConnectionManagerCallback-1919-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768661 INFO  (zkConnectionManagerCallback-1921-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768792 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 768793 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55777_solr
   [junit4]   2> 768794 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer Overseer (id=72133118504861699-127.0.0.1:55777_solr-n_0000000000) starting
   [junit4]   2> 768804 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 768804 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55777_solr
   [junit4]   2> 768804 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 768806 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 768806 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 768806 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:55777_solr]
   [junit4]   2> 768806 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 768806 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 768809 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 768809 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 768810 DEBUG (ScheduledTrigger-3340-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 768847 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55777.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 768855 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55777.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 768855 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55777.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 768856 INFO  (jetty-launcher-1910-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ComputePlanActionTest_D1A9C2BFDD39C093-001/tempDir-001/node1/.
   [junit4]   2> 768868 INFO  (zkConnectionManagerCallback-1925-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768871 INFO  (zkConnectionManagerCallback-1929-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768873 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 768874 INFO  (SUITE-ComputePlanActionTest-seed#[D1A9C2BFDD39C093]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:55775/solr ready
   [junit4]   2> 768885 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeWithMultipleReplicasLost
   [junit4]   2> 768886 DEBUG (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 768886 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 768887 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 768888 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 768889 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 768889 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 768902 INFO  (qtp442031494-8419) [n:127.0.0.1:55777_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 768904 DEBUG (qtp442031494-8415) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 768905 INFO  (qtp442031494-8415) [n:127.0.0.1:55777_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=12
   [junit4]   2> 768905 DEBUG (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 768905 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 768905 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 768906 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 768907 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 768908 INFO  (qtp442031494-8419) [n:127.0.0.1:55777_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 768909 DEBUG (qtp442031494-8420) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 768909 INFO  (qtp442031494-8420) [n:127.0.0.1:55777_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 768909 DEBUG (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 768909 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 768910 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 768910 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 768910 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 768911 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 768912 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 768912 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 768912 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 768912 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3cd5e833{/solr,null,AVAILABLE}
   [junit4]   2> 768912 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7d27ef02{HTTP/1.1,[http/1.1]}{127.0.0.1:55785}
   [junit4]   2> 768913 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.Server Started @770510ms
   [junit4]   2> 768913 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55785}
   [junit4]   2> 768913 ERROR (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 768913 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 768913 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 768913 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 768913 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T06:16:33.469632Z
   [junit4]   2> 768915 INFO  (zkConnectionManagerCallback-1932-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768916 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 768920 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55775/solr
   [junit4]   2> 768922 INFO  (zkConnectionManagerCallback-1936-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768925 INFO  (zkConnectionManagerCallback-1938-thread-1-processing-n:127.0.0.1:55785_solr) [n:127.0.0.1:55785_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 768935 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 768938 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 768941 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55785_solr
   [junit4]   2> 768942 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:55785_solr"} current state version: 0
   [junit4]   2> 768942 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:55785_solr
   [junit4]   2> 768943 INFO  (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 768943 INFO  (zkCallback-1928-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 768943 INFO  (zkCallback-1928-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 768944 INFO  (zkCallback-1937-thread-1-processing-n:127.0.0.1:55785_solr) [n:127.0.0.1:55785_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 768986 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55785.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 768997 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55785.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 768997 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55785.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 768998 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55785_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ComputePlanActionTest_D1A9C2BFDD39C093-001/tempDir-001/node2/.
   [junit4]   2> 769024 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 769025 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 769025 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 769025 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@35e215da{/solr,null,AVAILABLE}
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4f248542{HTTP/1.1,[http/1.1]}{127.0.0.1:55789}
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.Server Started @770623ms
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55789}
   [junit4]   2> 769026 ERROR (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 769026 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T06:16:33.582966Z
   [junit4]   2> 769028 INFO  (zkConnectionManagerCallback-1942-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769029 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 769035 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55775/solr
   [junit4]   2> 769036 INFO  (zkConnectionManagerCallback-1946-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769039 INFO  (zkConnectionManagerCallback-1948-thread-1-processing-n:127.0.0.1:55789_solr) [n:127.0.0.1:55789_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769049 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 769053 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 769056 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55789_solr
   [junit4]   2> 769057 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:55789_solr"} current state version: 0
   [junit4]   2> 769057 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:55789_solr
   [junit4]   2> 769058 INFO  (zkCallback-1937-thread-1-processing-n:127.0.0.1:55785_solr) [n:127.0.0.1:55785_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 769058 INFO  (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 769058 INFO  (zkCallback-1928-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 769058 INFO  (zkCallback-1928-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 769058 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:55789_solr) [n:127.0.0.1:55789_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 769097 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55789.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 769108 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55789.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 769109 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55789.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 769110 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55789_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ComputePlanActionTest_D1A9C2BFDD39C093-001/tempDir-001/node3/.
   [junit4]   2> 769147 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 769148 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 769148 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 769148 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 769148 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60d7762f{/solr,null,AVAILABLE}
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5404a2c2{HTTP/1.1,[http/1.1]}{127.0.0.1:55793}
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.e.j.s.Server Started @770746ms
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55793}
   [junit4]   2> 769149 ERROR (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 769149 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T06:16:33.705849Z
   [junit4]   2> 769151 INFO  (zkConnectionManagerCallback-1952-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769152 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 769158 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55775/solr
   [junit4]   2> 769159 INFO  (zkConnectionManagerCallback-1956-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769162 INFO  (zkConnectionManagerCallback-1958-thread-1-processing-n:127.0.0.1:55793_solr) [n:127.0.0.1:55793_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769173 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 769177 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 769180 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55793_solr
   [junit4]   2> 769181 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:55793_solr"} current state version: 0
   [junit4]   2> 769181 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:55793_solr
   [junit4]   2> 769182 INFO  (zkCallback-1928-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 769182 INFO  (zkCallback-1928-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 769182 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:55789_solr) [n:127.0.0.1:55789_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 769183 INFO  (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 769183 INFO  (zkCallback-1937-thread-1-processing-n:127.0.0.1:55785_solr) [n:127.0.0.1:55785_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 769183 INFO  (zkCallback-1957-thread-1-processing-n:127.0.0.1:55793_solr) [n:127.0.0.1:55793_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 769225 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55793.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 769242 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55793.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 769242 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55793.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@e0a1a80
   [junit4]   2> 769243 INFO  (TEST-ComputePlanActionTest.testNodeWithMultipleReplicasLost-seed#[D1A9C2BFDD39C093]) [n:127.0.0.1:55793_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ComputePlanActionTest_D1A9C2BFDD39C093-001/tempDir-001/node4/.
   [junit4]   2> 769290 INFO  (zkConnectionManagerCallback-1962-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 769295 INFO  (qtp442031494-8420) [n:127.0.0.1:55777_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769298 INFO  (qtp318701717-8452) [n:127.0.0.1:55785_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769299 INFO  (qtp27910889-8482) [n:127.0.0.1:55789_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769302 INFO  (qtp42324116-8502) [n:127.0.0.1:55793_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769302 DEBUG (qtp27910889-8478) [n:127.0.0.1:55789_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 769303 INFO  (qtp27910889-8478) [n:127.0.0.1:55789_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=9
   [junit4]   2> 769303 INFO  (qtp442031494-8417) [n:127.0.0.1:55777_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&collection.configName=conf&name=testNodeWithMultipleReplicasLost&nrtReplicas=3&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 769304 DEBUG (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 769305 DEBUG (zkCallback-1920-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:55777_solr, 127.0.0.1:55785_solr, 127.0.0.1:55789_solr, 127.0.0.1:55793_solr]
   [junit4]   2> 769305 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 769307 INFO  (OverseerThreadFactory-3342-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testNodeWithMultipleReplicasLost
   [junit4]   2> 769308 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 769309 DEBUG (OverseerAutoScalingTriggerThread-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 769309 DEBUG (ScheduledTrigger-3340-thread-1) [n:127.0.0.1:55777_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 4
   [junit4]   2> 769314 INFO  (qtp442031494-8420) [n:127.0.0.1:55777_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769316 INFO  (qtp318701717-8456) [n:127.0.0.1:55785_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769317 INFO  (qtp27910889-8482) [n:127.0.0.1:55789_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769318 INFO  (qtp42324116-8506) [n:127.0.0.1:55793_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 769328 INFO  (OverseerThreadFactory-3342-thread-1-processing-n:127.0.0.1:55777_solr) [n:127.0.0.1:55777_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 1152412706 sessionWrapper.createTime 1152412696312663, this.sessionWrapper.createTime 1152412696312663
   [junit4]   2> 769334 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "collection.configName":"conf",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 769334 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: testNodeWithMultipleReplicasLost
   [junit4]   2> 769335 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/testNodeWithMultipleReplicasLost/state.json
   [junit4]   2> 769444 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 769444 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 769445 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{}}
   [junit4]   2> 769445 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769447 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 769447 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 769448 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769448 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>       "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769450 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 769450 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 769451 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>       "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769452 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-ffffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>       "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node7":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard1_replica_n4",
   [junit4]   2>       "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769453 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 4, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 769453 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 769455 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{}}
   [junit4]   2> 769455 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator New Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node9":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>       "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769457 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 769457 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 769458 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node9":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>       "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769458 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator New Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node9":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>       "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node11":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>       "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769460 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 6, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 769460 INFO  (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 769461 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node9":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>       "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node11":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>       "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769461 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.SliceMutator New Slice: shard2:{
   [junit4]   2>   "range":"0-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node9":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>       "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node11":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>       "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node12":{
   [junit4]   2>       "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>       "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 769564 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/testNodeWithMultipleReplicasLost/state.json version: 0
   [junit4]   2> 769663 INFO  (qtp27910889-8482) [n:127.0.0.1:55789_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testNodeWithMultipleReplicasLost_shard1_replica_n1&action=CREATE&numShards=2&collection=testNodeWithMultipleReplicasLost&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 769663 INFO  (qtp442031494-8420) [n:127.0.0.1:55777_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testNodeWithMultipleReplicasLost_shard1_replica_n4&action=CREATE&numShards=2&collection=testNodeWithMultipleReplicasLost&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 769663 INFO  (qtp27910889-8482) [n:127.0.0.1:55789_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 769663 INFO  (qtp42324116-8506) [n:127.0.0.1:55793_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node9&collection.configName=conf&newCollection=true&name=testNodeWithMultipleReplicasLost_shard2_replica_n6&action=CREATE&numShards=2&collection=testNodeWithMultipleReplicasLost&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 769663 INFO  (qtp318701717-8456) [n:127.0.0.1:55785_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testNodeWithMultipleReplicasLost_shard1_replica_n2&action=CREATE&numShards=2&collection=testNodeWithMultipleReplicasLost&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 769663 INFO  (qtp442031494-8420) [n:127.0.0.1:55777_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 769663 INFO  (qtp42324116-8506) [n:127.0.0.1:55793_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 769664 INFO  (qtp318701717-8456) [n:127.0.0.1:55785_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 769664 INFO  (qtp442031494-8421) [n:127.0.0.1:55777_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node11&collection.configName=conf&newCollection=true&name=testNodeWithMultipleReplicasLost_shard2_replica_n8&action=CREATE&numShards=2&collection=testNodeWithMultipleReplicasLost&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 769665 INFO  (qtp318701717-8457) [n:127.0.0.1:55785_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node12&collection.configName=conf&newCollection=true&name=testNodeWithMultipleReplicasLost_shard2_replica_n10&action=CREATE&numShards=2&collection=testNodeWithMultipleReplicasLost&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 769672 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>   "core_node_name":"core_node12",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>   "core_node_name":"core_node12",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node12:{"core":"testNodeWithMultipleReplicasLost_shard2_replica_n10","base_url":"http://127.0.0.1:55785/solr","node_name":"127.0.0.1:55785_solr","state":"down","type":"NRT"}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(testNodeWithMultipleReplicasLost//collections/testNodeWithMultipleReplicasLost/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node7":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n4",
   [junit4]   2>           "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"}}},
   [junit4]   2>     "shard2":{
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node9":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>           "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node11":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>           "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node12":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>           "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>           "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"}}}}}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>   "core_node_name":"core_node5",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>   "core_node_name":"core_node5",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node5:{"core":"testNodeWithMultipleReplicasLost_shard1_replica_n2","base_url":"http://127.0.0.1:55785/solr","node_name":"127.0.0.1:55785_solr","state":"down","type":"NRT"}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(testNodeWithMultipleReplicasLost//collections/testNodeWithMultipleReplicasLost/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>           "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node7":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n4",
   [junit4]   2>           "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"}}},
   [junit4]   2>     "shard2":{
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node9":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>           "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node11":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>           "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node12":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>           "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>           "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"}}}}}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>   "core_node_name":"core_node11",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55777_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>   "core_node_name":"core_node11",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55777_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node11:{"core":"testNodeWithMultipleReplicasLost_shard2_replica_n8","base_url":"http://127.0.0.1:55777/solr","node_name":"127.0.0.1:55777_solr","state":"down","type":"NRT"}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(testNodeWithMultipleReplicasLost//collections/testNodeWithMultipleReplicasLost/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{
   [junit4]   2>     "shard1":{
   [junit4]   2>       "range":"80000000-ffffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>           "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node7":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard1_replica_n4",
   [junit4]   2>           "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"}}},
   [junit4]   2>     "shard2":{
   [junit4]   2>       "range":"0-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node9":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n6",
   [junit4]   2>           "base_url":"http://127.0.0.1:55793/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node11":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n8",
   [junit4]   2>           "base_url":"http://127.0.0.1:55777/solr",
   [junit4]   2>           "node_name":"127.0.0.1:55777_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node12":{
   [junit4]   2>           "core":"testNodeWithMultipleReplicasLost_shard2_replica_n10",
   [junit4]   2>           "base_url":"http://127.0.0.1:55785/solr",
   [junit4]   2>           "node_name":"127.0.0.1:55785_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"}}}}}
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55789_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 769673 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"testNodeWithMultipleReplicasLost_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:55789/solr",
   [junit4]   2>   "node_name":"127.0.0.1:55789_solr",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"testNodeWithMultipleReplicasLost",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 769674 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node3:{"core":"testNodeWithMultipleReplicasLost_shard1_replica_n1","base_url":"http://127.0.0.1:55789/solr","node_name":"127.0.0.1:55789_solr","state":"down","type":"NRT"}
   [junit4]   2> 769674 DEBUG (OverseerStateUpdate-72133118504861699-127.0.0.1:55777_solr-n_0000000000) [n:127.0.0.1:55777_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(testNodeWithMultipleReplicasLost//collections/testNodeWithMultipleReplicasLo

[...truncated too long message...]

82-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@761eeb43{/solr,null,UNAVAILABLE}
   [junit4]   2> 130273 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10044e435be0005, likely client has closed socket
   [junit4]   2> 130274 INFO  (jetty-closer-482-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 130274 INFO  (jetty-closer-482-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@620e5beb{/solr,null,UNAVAILABLE}
   [junit4]   2> 130274 INFO  (jetty-closer-482-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 160279 WARN  (jetty-closer-482-thread-1) [    ] o.e.j.u.t.QueuedThreadPool QueuedThreadPool@qtp343045489{STOPPING,8<=9<=10000,i=0,q=1} Couldn't stop Thread[qtp343045489-1811,5,TGRP-JdbcTest]
   [junit4]   2> 160280 ERROR (SUITE-JdbcTest-seed#[DE1A845813A31629]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 160280 INFO  (SUITE-JdbcTest-seed#[DE1A845813A31629]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54777 54777
   [junit4]   2> 160310 INFO  (Thread-262) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54777 54777
   [junit4]   2> 160543 WARN  (Thread-262) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 167 /solr/aliases.json
   [junit4]   2> 167 /solr/clusterprops.json
   [junit4]   2> 2 /solr/security.json
   [junit4]   2> 2 /solr/configs/conf
   [junit4]   2> 2 /solr/collections/collection1_collection/terms/shard1
   [junit4]   2> 2 /solr/collections/collection1_collection/terms/shard2
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 167 /solr/clusterstate.json
   [junit4]   2> 12 /solr/collections/collection1_collection/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 167 /solr/live_nodes
   [junit4]   2> 167 /solr/collections
   [junit4]   2>
   [junit4]   2> 160543 INFO  (SUITE-JdbcTest-seed#[DE1A845813A31629]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Feb 03, 2018 7:13:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Feb 03, 2018 7:14:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.sql.JdbcTest:
   [junit4]   2>    1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
   [junit4]   2>         at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.base@9/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> Feb 03, 2018 7:14:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
   [junit4]   2> Feb 03, 2018 7:14:10 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
   [junit4]   2>         at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.base@9/java.lang.Thread.run(Thread.java:844)
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@47936625), locale=es-GT, timezone=Europe/Stockholm
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=2,free=95760720,total=314572800
   [junit4]   2> NOTE: All tests run in this JVM: [SolrExampleJettyTest, JDBCStreamTest, SolrExampleBinaryTest, SquareRootEvaluatorTest, TestSuggesterResponse, LBHttpSolrClientBuilderTest, TestNamedListCodec, ClientUtilsTest, NormalizeEvaluatorTest, TangentEvaluatorTest, TestConfigSetAdminRequest, ArcCosineEvaluatorTest, FieldValueEvaluatorTest, StreamExpressionParserTest, OrEvaluatorTest, RegressionEvaluatorTest, CubedRootEvaluatorTest, CloudSolrClientCacheTest, OperationsTest, SolrParamTest, TestPathTrie, GreaterThanEqualToEvaluatorTest, CollectionAdminRequestRequiredParamsTest, SolrSchemalessExampleTest, CoalesceEvaluatorTest, SolrZkClientTest, ModuloEvaluatorTest, TestZkConfigManager, StreamExpressionToExplanationTest, CloudSolrClientTest, SolrQueryTest, UniformDistributionEvaluatorTest, AscEvaluatorTest, GraphExpressionTest, LargeVolumeBinaryJettyTest, SolrExampleStreamingTest, SolrExampleStreamingBinaryTest, SolrExampleEmbeddedTest, TestJavaBinCodec, TestHash, ContentStreamTest, TestFastInputStream, DocumentAnalysisResponseTest, SolrExceptionTest, AnlysisResponseBaseTest, UsingSolrJRefGuideExamplesTest, TestPolicy, BasicHttpSolrClientTest, HttpSolrClientBuilderTest, HttpSolrClientConPoolTest, HttpSolrClientSSLAuthConPoolTest, GraphTest, JdbcDriverTest, JdbcTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=JdbcTest -Dtests.seed=DE1A845813A31629 -Dtests.slow=true -Dtests.locale=es-GT -Dtests.timezone=Europe/Stockholm -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | JdbcTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.sql.JdbcTest:
   [junit4]    >    1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
   [junit4]    >         at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.base@9/java.lang.Thread.run(Thread.java:844)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DE1A845813A31629]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=1811, name=qtp343045489-1811, state=TIMED_WAITING, group=TGRP-JdbcTest]
   [junit4]    >         at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.base@9/java.lang.Thread.run(Thread.java:844)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DE1A845813A31629]:0)
   [junit4] Completed [106/146 (1!)] on J1 in 63.13s, 10 tests, 2 errors, 1 skipped <<< FAILURES!

[...truncated 40054 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4423 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4423/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventFromRestoredState

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([AC315A153E740B32:AC07EED51779AF58]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventFromRestoredState(TriggerIntegrationTest.java:760)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13094 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/init-core-data-001
   [junit4]   2> 1791710 WARN  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1791710 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1791715 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1791715 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1791715 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001
   [junit4]   2> 1791716 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1791716 INFO  (Thread-2929) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1791716 INFO  (Thread-2929) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1791719 ERROR (Thread-2929) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1791826 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:65180
   [junit4]   2> 1791837 INFO  (zkConnectionManagerCallback-4018-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791841 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1791841 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1791842 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1791843 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1791843 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79d4e366{/solr,null,AVAILABLE}
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@c692d84{HTTP/1.1,[http/1.1]}{127.0.0.1:65182}
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1aadb231{/solr,null,AVAILABLE}
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-1) [    ] o.e.j.s.Server Started @1795621ms
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65182}
   [junit4]   2> 1791844 ERROR (jetty-launcher-4015-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6481cbc{HTTP/1.1,[http/1.1]}{127.0.0.1:65183}
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.e.j.s.Server Started @1795621ms
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1791844 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65183}
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T16:56:09.800Z
   [junit4]   2> 1791845 ERROR (jetty-launcher-4015-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1791845 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T16:56:09.800Z
   [junit4]   2> 1791847 INFO  (zkConnectionManagerCallback-4020-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791847 INFO  (zkConnectionManagerCallback-4022-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791847 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1791847 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1791857 INFO  (jetty-launcher-4015-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65180/solr
   [junit4]   2> 1791858 INFO  (jetty-launcher-4015-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65180/solr
   [junit4]   2> 1791859 INFO  (zkConnectionManagerCallback-4028-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791859 INFO  (zkConnectionManagerCallback-4030-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791862 INFO  (zkConnectionManagerCallback-4034-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791862 INFO  (zkConnectionManagerCallback-4033-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1792005 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1792005 INFO  (jetty-launcher-4015-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1792005 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65183_solr
   [junit4]   2> 1792007 INFO  (jetty-launcher-4015-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65182_solr
   [junit4]   2> 1792007 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.Overseer Overseer (id=72135633553063941-127.0.0.1:65183_solr-n_0000000000) starting
   [junit4]   2> 1792009 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1792009 INFO  (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1792019 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1792020 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1792021 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65182_solr]
   [junit4]   2> 1792021 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1792021 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1792022 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65183_solr
   [junit4]   2> 1792025 INFO  (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1792025 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1792026 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1792027 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1792027 DEBUG (ScheduledTrigger-6069-thread-1) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1792133 INFO  (jetty-launcher-4015-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65182.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792135 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65183.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792142 INFO  (jetty-launcher-4015-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65182.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792142 INFO  (jetty-launcher-4015-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65182.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792142 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65183.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792142 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65183.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792143 INFO  (jetty-launcher-4015-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001/node1/.
   [junit4]   2> 1792143 INFO  (jetty-launcher-4015-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001/node2/.
   [junit4]   2> 1792182 INFO  (zkConnectionManagerCallback-4040-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1792186 INFO  (zkConnectionManagerCallback-4044-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1792188 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1792189 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:65180/solr ready
   [junit4]   2> 1792208 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 1792208 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 1792209 DEBUG (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1792209 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1792211 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1792212 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1792212 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1792218 INFO  (qtp495146565-16886) [n:127.0.0.1:65182_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1792233 INFO  (qtp495146565-16886) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=15
   [junit4]   2> 1792237 INFO  (qtp495146565-16892) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1792240 INFO  (qtp1622839406-16891) [n:127.0.0.1:65183_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1792240 DEBUG (qtp495146565-16888) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1792241 INFO  (qtp495146565-16888) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 1792241 DEBUG (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1792242 INFO  (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 1792242 DEBUG (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65182_solr, 127.0.0.1:65183_solr]
   [junit4]   2> 1792242 DEBUG (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEQ instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 1792242 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1792244 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1792245 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1792245 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1792246 DEBUG (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1792246 DEBUG (ScheduledTrigger-6069-thread-1) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1792246 DEBUG (ScheduledTrigger-6069-thread-1) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1792247 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1792247 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1792247 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1792247 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6ec4e2e9{/solr,null,AVAILABLE}
   [junit4]   2> 1792262 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7de9dc27{HTTP/1.1,[http/1.1]}{127.0.0.1:65195}
   [junit4]   2> 1792262 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server Started @1796039ms
   [junit4]   2> 1792262 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65195}
   [junit4]   2> 1792262 ERROR (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1792262 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1792262 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1792262 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1792263 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T16:56:10.218Z
   [junit4]   2> 1792264 INFO  (zkConnectionManagerCallback-4046-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1792265 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1792274 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65180/solr
   [junit4]   2> 1792275 INFO  (zkConnectionManagerCallback-4050-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1792278 INFO  (zkConnectionManagerCallback-4052-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1792289 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1792292 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1792296 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65195_solr
   [junit4]   2> 1792298 INFO  (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1792298 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1792298 INFO  (zkCallback-4043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1792299 INFO  (zkCallback-4051-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1792389 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65195.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792432 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65195.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792432 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65195.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1792433 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [n:127.0.0.1:65195_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001/node3/.
   [junit4]   2> 1793250 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1793250 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1793250 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:65195_solr at time 1190790032038989
   [junit4]   2> 1794222 INFO  (OverseerCollectionConfigSetProcessor-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1794255 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1794255 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1794255 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEQ firing registered processor for nodes: [127.0.0.1:65195_solr] added at times [1190790032038989], now=1190791037604043
   [junit4]   2> 1794256 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1190790032038989,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190790032038989],
   [junit4]   2>     "nodeNames":["127.0.0.1:65195_solr"]}}
   [junit4]   2> 1794256 DEBUG (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerEQ]
   [junit4]   2> 1794258 INFO  (ScheduledTrigger-6069-thread-2) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1794259 DEBUG (AutoscalingActionExecutor-6070-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1190790032038989,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190790032038989],
   [junit4]   2>     "_enqueue_time_":1190791038189150,
   [junit4]   2>     "nodeNames":["127.0.0.1:65195_solr"]}}
   [junit4]   2> 1794259 INFO  (AutoscalingActionExecutor-6070-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.TriggerIntegrationTest -- event: {
   [junit4]   2>   "id":"43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1190790032038989,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190790032038989],
   [junit4]   2>     "_enqueue_time_":1190791038189150,
   [junit4]   2>     "nodeNames":["127.0.0.1:65195_solr"]}}
   [junit4]   2> 1794260 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6481cbc{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1794261 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=340574863
   [junit4]   2> 1794261 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1794261 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6193a7f3: rootName = solr_65183, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@40eef977
   [junit4]   2> 1794268 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1794268 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6b266690: rootName = solr_65183, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@17c7e1bc
   [junit4]   2> 1794272 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1794272 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7b337e41: rootName = solr_65183, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5993827d
   [junit4]   2> 1794275 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:65183_solr
   [junit4]   2> 1794277 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1794277 INFO  (zkCallback-4032-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1794277 INFO  (zkCallback-4043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1794277 INFO  (zkCallback-4051-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 1794277 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1794278 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1794278 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.Overseer Overseer (id=72135633553063941-127.0.0.1:65183_solr-n_0000000000) closing
   [junit4]   2> 1794278 INFO  (OverseerStateUpdate-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:65183_solr
   [junit4]   2> 1794279 WARN  (OverseerAutoScalingTriggerThread-72135633553063941-127.0.0.1:65183_solr-n_0000000000) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1794279 DEBUG (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1794279 WARN  (AutoscalingActionExecutor-6070-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_added_triggerEQ has been closed.
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.dequeue(ScheduledTriggers.java:525)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1794280 DEBUG (AutoscalingActionExecutor-6070-thread-1-processing-n:127.0.0.1:65183_solr) [n:127.0.0.1:65183_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 20 ms for event id=43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9
   [junit4]   2> 1794280 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65182_solr
   [junit4]   2> 1794281 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1aadb231{/solr,null,UNAVAILABLE}
   [junit4]   2> 1794281 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1794282 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.Overseer Overseer (id=72135633553063942-127.0.0.1:65182_solr-n_0000000001) starting
   [junit4]   2> 1794298 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1794303 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1794303 INFO  (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 1794303 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65182_solr, 127.0.0.1:65195_solr]
   [junit4]   2> 1794303 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEQ instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 1794304 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65182_solr, 127.0.0.1:65195_solr]
   [junit4]   2> 1794304 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
   [junit4]   2> 1794304 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1794308 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1794309 DEBUG (ScheduledTrigger-6084-thread-1) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1794309 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1794310 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1190790032038989,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190790032038989],
   [junit4]   2>     "_dequeue_time_":1190791092071024,
   [junit4]   2>     "_enqueue_time_":1190791038189150,
   [junit4]   2>     "nodeNames":["127.0.0.1:65195_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1794310 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_triggerEQ]
   [junit4]   2> 1794311 INFO  (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1794311 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1190790032038989,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190790032038989],
   [junit4]   2>     "_dequeue_time_":1190791092071024,
   [junit4]   2>     "_enqueue_time_":1190791038189150,
   [junit4]   2>     "nodeNames":["127.0.0.1:65195_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1794311 INFO  (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.TriggerIntegrationTest -- event: {
   [junit4]   2>   "id":"43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1190790032038989,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190790032038989],
   [junit4]   2>     "_dequeue_time_":1190791092071024,
   [junit4]   2>     "_enqueue_time_":1190791038189150,
   [junit4]   2>     "nodeNames":["127.0.0.1:65195_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1794312 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1794312 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1794313 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 5000ms
   [junit4]   2> 1794313 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerEQ after 5000ms
   [junit4]   2> 1794313 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=43b0468dc9c4dT2u8fpfce50bfve5huh8xetey9
   [junit4]   2> 1799284 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> 1799297 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 1799298 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
   [junit4]   2> 1799299 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1799299 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1799302 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1799303 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1799303 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1799311 INFO  (qtp378882778-16950) [n:127.0.0.1:65195_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1799313 INFO  (qtp495146565-16888) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1799314 DEBUG (qtp495146565-16890) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1799315 INFO  (qtp495146565-16890) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 1799315 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1799316 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65182_solr, 127.0.0.1:65195_solr]
   [junit4]   2> 1799316 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1799316 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1799318 INFO  (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1799319 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1799319 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1799319 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1799320 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1799320 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1799320 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1799320 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1799320 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1799321 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7918a30b{/solr,null,AVAILABLE}
   [junit4]   2> 1799321 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@69c9f7fb{HTTP/1.1,[http/1.1]}{127.0.0.1:65210}
   [junit4]   2> 1799321 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server Started @1803098ms
   [junit4]   2> 1799321 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65210}
   [junit4]   2> 1799321 ERROR (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1799322 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1799322 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1799322 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1799322 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T16:56:17.277Z
   [junit4]   2> 1799324 INFO  (zkConnectionManagerCallback-4056-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1799324 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1799335 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65180/solr
   [junit4]   2> 1799336 INFO  (zkConnectionManagerCallback-4060-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1799340 INFO  (zkConnectionManagerCallback-4062-thread-1-processing-n:127.0.0.1:65210_solr) [n:127.0.0.1:65210_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1799350 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1799352 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1799356 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65210_solr
   [junit4]   2> 1799358 INFO  (zkCallback-4051-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1799358 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1799358 INFO  (zkCallback-4043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1799358 INFO  (zkCallback-4061-thread-1-processing-n:127.0.0.1:65210_solr) [n:127.0.0.1:65210_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1799429 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65210.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1799438 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65210.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1799438 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65210.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1799440 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [n:127.0.0.1:65210_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001/node4/.
   [junit4]   2> 1800326 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1800326 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1800326 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:65210_solr at time 1190797107959578
   [junit4]   2> 1801334 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1801334 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1802335 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1802335 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1802335 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:65210_solr] added at times [1190797107959578], now=1190799117254161
   [junit4]   2> 1802335 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"43b060e9e971aT2u8fpfce50bfve5huh8xeteyc",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1190797107959578,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190797107959578],
   [junit4]   2>     "nodeNames":["127.0.0.1:65210_solr"]}}
   [junit4]   2> 1802335 DEBUG (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger]
   [junit4]   2> 1802336 INFO  (ScheduledTrigger-6084-thread-3) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1802336 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"43b060e9e971aT2u8fpfce50bfve5huh8xeteyc",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1190797107959578,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190797107959578],
   [junit4]   2>     "_enqueue_time_":1190799117466845,
   [junit4]   2>     "nodeNames":["127.0.0.1:65210_solr"]}}
   [junit4]   2> 1802337 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 5000ms
   [junit4]   2> 1802337 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 1 ms for event id=43b060e9e971aT2u8fpfce50bfve5huh8xeteyc
   [junit4]   2> 1802361 INFO  (qtp495146565-16890) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1802366 INFO  (qtp378882778-16952) [n:127.0.0.1:65195_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1802369 INFO  (qtp1626726393-16987) [n:127.0.0.1:65210_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1802369 DEBUG (qtp495146565-16892) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1802370 INFO  (qtp495146565-16892) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=24
   [junit4]   2> 1802370 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1802371 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:65210_solr, 127.0.0.1:65182_solr, 127.0.0.1:65195_solr]
   [junit4]   2> 1802371 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1802371 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1802371 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1802372 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1804378 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4]   2> 1804396 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 1804396 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 8
   [junit4]   2> 1804397 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 1804397 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1804398 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1804399 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1804399 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1804399 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1804400 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1804400 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1804400 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1804401 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1872c1e6{/solr,null,AVAILABLE}
   [junit4]   2> 1804401 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@225ac740{HTTP/1.1,[http/1.1]}{127.0.0.1:65215}
   [junit4]   2> 1804401 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server Started @1808178ms
   [junit4]   2> 1804401 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65215}
   [junit4]   2> 1804401 ERROR (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1804401 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1804402 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1804402 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1804402 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T16:56:22.357Z
   [junit4]   2> 1804403 INFO  (zkConnectionManagerCallback-4066-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1804404 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1804413 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65180/solr
   [junit4]   2> 1804414 INFO  (zkConnectionManagerCallback-4070-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1804417 INFO  (zkConnectionManagerCallback-4072-thread-1-processing-n:127.0.0.1:65215_solr) [n:127.0.0.1:65215_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1804427 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1804430 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1804433 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65215_solr
   [junit4]   2> 1804438 INFO  (zkCallback-4051-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1804438 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1804438 INFO  (zkCallback-4061-thread-1-processing-n:127.0.0.1:65210_solr) [n:127.0.0.1:65210_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1804438 INFO  (zkCallback-4043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1804439 INFO  (zkCallback-4071-thread-1-processing-n:127.0.0.1:65215_solr) [n:127.0.0.1:65215_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1804506 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65215.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1804514 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65215.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1804514 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65215.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1804516 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [n:127.0.0.1:65215_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001/node5/.
   [junit4]   2> 1804576 INFO  (qtp495146565-16892) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1804578 INFO  (qtp378882778-16952) [n:127.0.0.1:65195_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1804579 INFO  (qtp1626726393-16992) [n:127.0.0.1:65210_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1804582 INFO  (qtp421096667-17012) [n:127.0.0.1:65215_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1804582 DEBUG (qtp1626726393-16991) [n:127.0.0.1:65210_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1804583 INFO  (qtp1626726393-16991) [n:127.0.0.1:65210_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=8
   [junit4]   2> 1804583 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 1804584 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65210_solr, 127.0.0.1:65215_solr, 127.0.0.1:65182_solr, 127.0.0.1:65195_solr]
   [junit4]   2> 1804584 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1804584 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@225ac740{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1804584 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1149956601
   [junit4]   2> 1804585 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1804585 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@46c4994e: rootName = solr_65215, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5056e907
   [junit4]   2> 1804587 INFO  (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1804587 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1804588 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1804588 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 4
   [junit4]   2> 1804593 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1804593 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@369c3cbb: rootName = solr_65215, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@17c7e1bc
   [junit4]   2> 1804597 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1804597 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2488d349: rootName = solr_65215, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5993827d
   [junit4]   2> 1804599 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:65215_solr
   [junit4]   2> 1804600 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1804600 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1804601 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1804601 INFO  (zkCallback-4061-thread-1-processing-n:127.0.0.1:65210_solr) [n:127.0.0.1:65210_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1804601 INFO  (zkCallback-4051-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1804601 INFO  (zkCallback-4043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1804601 INFO  (zkCallback-4071-thread-1-processing-n:127.0.0.1:65215_solr) [n:127.0.0.1:65215_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1804603 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1872c1e6{/solr,null,UNAVAILABLE}
   [junit4]   2> 1804603 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1805599 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
   [junit4]   2> 1805599 DEBUG (ScheduledTrigger-6084-thread-2) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:65215_solr
   [junit4]   2> 1806124 INFO  (qtp495146565-16892) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1806127 INFO  (qtp378882778-16954) [n:127.0.0.1:65195_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1806130 INFO  (qtp1626726393-16989) [n:127.0.0.1:65210_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1806130 DEBUG (qtp495146565-16888) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1806131 INFO  (qtp495146565-16888) [n:127.0.0.1:65182_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=10
   [junit4]   2> 1806131 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 1806132 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:65210_solr, 127.0.0.1:65182_solr, 127.0.0.1:65195_solr]
   [junit4]   2> 1806132 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 1806133 INFO  (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1806134 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:65215_solr
   [junit4]   2> 1806134 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1806135 DEBUG (ScheduledTrigger-6084-thread-4) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
   [junit4]   2> 1806135 DEBUG (ScheduledTrigger-6084-thread-4) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:65215_solr]
   [junit4]   2> 1806135 DEBUG (ScheduledTrigger-6084-thread-4) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"43b0768cc9be1T2u8fpfce50bfve5huh8xeteye",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":1190802915892193,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190802915892193],
   [junit4]   2>     "nodeNames":["127.0.0.1:65215_solr"]}}
   [junit4]   2> 1806135 DEBUG (ScheduledTrigger-6084-thread-4) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_restore_trigger]
   [junit4]   2> 1806135 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 1806136 INFO  (ScheduledTrigger-6084-thread-4) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1806136 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"43b0768cc9be1T2u8fpfce50bfve5huh8xeteye",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":1190802915892193,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1190802915892193],
   [junit4]   2>     "_enqueue_time_":1190802917044991,
   [junit4]   2>     "nodeNames":["127.0.0.1:65215_solr"]}}
   [junit4]   2> 1806136 INFO  (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 1806144 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_restore_trigger after 5000ms
   [junit4]   2> 1806145 DEBUG (AutoscalingActionExecutor-6085-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 9 ms for event id=43b0768cc9be1T2u8fpfce50bfve5huh8xeteye
   [junit4]   2> 1806159 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 1806159 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 11
   [junit4]   2> 1806160 DEBUG (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 1806160 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 1806162 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1806162 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1806163 DEBUG (OverseerAutoScalingTriggerThread-72135633553063942-127.0.0.1:65182_solr-n_0000000001) [n:127.0.0.1:65182_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 1806163 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1806164 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1806164 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1806164 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28ddbfa2{/solr,null,AVAILABLE}
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2bdf023{HTTP/1.1,[http/1.1]}{127.0.0.1:65224}
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server Started @1809942ms
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65224}
   [junit4]   2> 1806165 ERROR (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1806165 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-03T16:56:24.120Z
   [junit4]   2> 1806167 INFO  (zkConnectionManagerCallback-4076-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1806168 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1806177 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65180/solr
   [junit4]   2> 1806178 INFO  (zkConnectionManagerCallback-4080-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1806181 INFO  (zkConnectionManagerCallback-4082-thread-1-processing-n:127.0.0.1:65224_solr) [n:127.0.0.1:65224_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1806190 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1806193 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1806196 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65224_solr
   [junit4]   2> 1806206 INFO  (zkCallback-4061-thread-1-processing-n:127.0.0.1:65210_solr) [n:127.0.0.1:65210_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1806206 INFO  (zkCallback-4031-thread-1-processing-n:127.0.0.1:65182_solr) [n:127.0.0.1:65182_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1806206 INFO  (zkCallback-4051-thread-1-processing-n:127.0.0.1:65195_solr) [n:127.0.0.1:65195_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1806207 INFO  (zkCallback-4043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1806207 INFO  (zkCallback-4081-thread-1-processing-n:127.0.0.1:65224_solr) [n:127.0.0.1:65224_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1806267 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65224.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1806295 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65224.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1806296 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65224.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@456f8f7c
   [junit4]   2> 1806298 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [n:127.0.0.1:65224_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001/tempDir-001/node6/.
   [junit4]   2> 1806345 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T12:27:37-09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1806347 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1806347 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1806347 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1806347 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18f0b5fd{/solr,null,AVAILABLE}
   [junit4]   2> 1806348 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3745be70{HTTP/1.1,[http/1.1]}{127.0.0.1:65228}
   [junit4]   2> 1806348 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.e.j.s.Server Started @1810125ms
   [junit4]   2> 1806348 INFO  (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A153E740B32]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65228}
   [junit4]   2> 1806348 ERROR (TEST-TriggerIntegrationTest.testSearchRate-seed#[AC315A15

[...truncated too long message...]

$LiveNodeWatcher.process(ZkStateReader.java:1085)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:269)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1984686 WARN  (zkCallback-4147-thread-4-processing-n:127.0.0.1:65352_solr) [n:127.0.0.1:65352_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1984687 INFO  (jetty-closer-4016-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7c930110{/solr,null,UNAVAILABLE}
   [junit4]   2> 1984687 INFO  (jetty-closer-4016-thread-7) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1984689 ERROR (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1984690 INFO  (SUITE-TriggerIntegrationTest-seed#[AC315A153E740B32]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:65180 65180
   [junit4]   2> 1984734 INFO  (Thread-2929) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:65180 65180
   [junit4]   2> 1985022 WARN  (Thread-2929) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 39 /solr/collections/collection1/terms/shard1
   [junit4]   2> 21 /solr/clusterprops.json
   [junit4]   2> 21 /solr/aliases.json
   [junit4]   2> 20 /solr/security.json
   [junit4]   2> 9 /solr/collections/testMetricTrigger/terms/shard1
   [junit4]   2> 9 /solr/configs/conf
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard2
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 147 /solr/collections/collection1/state.json
   [junit4]   2> 45 /solr/collections/testMetricTrigger/state.json
   [junit4]   2> 21 /solr/clusterstate.json
   [junit4]   2> 5 /solr/autoscaling.json
   [junit4]   2> 5 /solr/overseer_elect/election/72135633553063962-127.0.0.1:65232_solr-n_0000000007
   [junit4]   2> 3 /solr/overseer_elect/election/72135633553063977-127.0.0.1:65370_solr-n_0000000012
   [junit4]   2> 2 /solr/overseer_elect/election/72135633553063974-127.0.0.1:65352_solr-n_0000000011
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72135633553063956-core_node8-n_0000000003
   [junit4]   2> 2 /solr/overseer_elect/election/72135633553063992-127.0.0.1:49268_solr-n_0000000017
   [junit4]   2> 2 /solr/overseer_elect/election/72135633553063968-127.0.0.1:65322_solr-n_0000000009
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72135633553063959-core_node4-n_0000000001
   [junit4]   2> 2 /solr/overseer_elect/election/72135633553063959-127.0.0.1:65228_solr-n_0000000006
   [junit4]   2> 2 /solr/overseer_elect/election/72135633553063950-127.0.0.1:65210_solr-n_0000000003
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 21 /solr/live_nodes
   [junit4]   2> 21 /solr/collections
   [junit4]   2> 5 /solr/overseer/queue-work
   [junit4]   2> 5 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 5 /solr/overseer/queue
   [junit4]   2> 5 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/autoscaling/events/node_lost_triggerMR
   [junit4]   2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_AC315A153E740B32-001
   [junit4]   2> Feb 03, 2018 4:59:23 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=Lucene50(blocksize=128), x_s=PostingsFormat(name=MockRandom)}, docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=635, maxMBSortInHeap=6.953536986137706, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@25700a32), locale=hr, timezone=Asia/Dubai
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=179803624,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [AnalyticsQueryTest, FileBasedSpellCheckerTest, TestLegacyTerms, TestCloudManagedSchema, TestLuceneIndexBackCompat, HdfsSyncSliceTest, CollectionsAPIAsyncDistributedZkTest, ParsingFieldUpdateProcessorsTest, ChaosMonkeySafeLeaderWithPullReplicasTest, ExternalFileFieldSortTest, TestStressUserVersions, TestDynamicLoading, CoreAdminHandlerTest, RegexBoostProcessorTest, VMParamsZkACLAndCredentialsProvidersTest, DeleteStatusTest, PrimUtilsTest, DocValuesNotIndexedTest, ZkStateWriterTest, TestNodeLostTrigger, TestRandomFaceting, TestManagedResourceStorage, TestConfig, FacetPivotSmallTest, SimpleFacetsTest, TestSolr4Spatial, TestSchemaSimilarityResource, SolrSlf4jReporterTest, BufferStoreTest, VersionInfoTest, CSVRequestHandlerTest, DefaultValueUpdateProcessorTest, TestSolrQueryParser, TestSolrConfigHandlerConcurrent, TestRandomCollapseQParserPlugin, TestManagedSchemaThreadSafety, DeleteLastCustomShardedReplicaTest, PrimitiveFieldTypeTest, ScriptEngineTest, TolerantUpdateProcessorTest, TestSolrConfigHandler, TestSearchPerf, TestShardHandlerFactory, HttpPartitionTest, TestLMDirichletSimilarityFactory, TestUseDocValuesAsStored2, TestRawTransformer, CurrencyFieldTypeTest, JSONWriterTest, SoftAutoCommitTest, CollectionReloadTest, TestPullReplicaErrorHandling, RuleEngineTest, ShowFileRequestHandlerTest, TestComplexPhraseQParserPlugin, SortByFunctionTest, URLClassifyProcessorTest, FieldAnalysisRequestHandlerTest, TestIndexingPerformance, CdcrBootstrapTest, TestNRTOpen, AsyncCallRequestStatusResponseTest, TestClassicSimilarityFactory, TestInPlaceUpdatesStandalone, LeaderFailureAfterFreshStartTest, TestSchemaNameResource, TestCloudSearcherWarming, NotRequiredUniqueKeyTest, SegmentsInfoRequestHandlerTest, TestComponentsName, BasicDistributedZk2Test, SimpleCollectionCreateDeleteTest, ResponseHeaderTest, TestSha256AuthenticationProvider, CheckHdfsIndexTest, OutOfBoxZkACLAndCredentialsProvidersTest, UninvertDocValuesMergePolicyTest, SolrCmdDistributorTest, TestWriterPerf, SchemaApiFailureTest, DistribDocExpirationUpdateProcessorTest, TestPivotHelperCode, NodeLostTriggerTest, TestCollectionsAPIViaSolrCloudCluster, TestOrdValues, TestOverriddenPrefixQueryForCustomFieldType, PropertiesRequestHandlerTest, TestLeaderElectionWithEmptyReplica, OverseerTest, TestSortableTextField, TestReversedWildcardFilterFactory, CdcrVersionReplicationTest, TestCloudPseudoReturnFields, TestSimDistributedQueue, TestSubQueryTransformer, JavabinLoaderTest, TestDistribIDF, TestJettySolrRunner, TestNumericRangeQuery32, TestDistributedSearch, ConnectionManagerTest, TestSolrFieldCacheBean, SolrCLIZkUtilsTest, RequestLoggingTest, ResponseLogComponentTest, TestLegacyFieldReuse, TestRequestStatusCollectionAPI, SpellCheckCollatorTest, TestConfigOverlay, TestZkAclsWithHadoopAuth, V2ApiIntegrationTest, TestFuzzyAnalyzedSuggestions, TestSimpleQParserPlugin, TestRTimerTree, CursorMarkTest, CollectionStateFormat2Test, PKIAuthenticationIntegrationTest, OverseerRolesTest, SparseHLLTest, PeerSyncTest, BlobRepositoryCloudTest, JvmMetricsTest, TestGraphMLResponseWriter, ComputePlanActionTest, TestReloadDeadlock, ReplicaListTransformerTest, TestDocumentBuilder, TestLegacyFieldCache, DistributedFacetPivotSmallAdvancedTest, DistributedExpandComponentTest, TestFieldTypeCollectionResource, SpellPossibilityIteratorTest, DistributedSpellCheckComponentTest, ExitableDirectoryReaderTest, BigEndianAscendingWordSerializerTest, TestFastLRUCache, TestImpersonationWithHadoopAuth, TestPushWriter, TestCoreDiscovery, TestRandomDVFaceting, IndexBasedSpellCheckerTest, ClassificationUpdateProcessorFactoryTest, CreateRoutedAliasTest, TestLocalFSCloudBackupRestore, CdcrUpdateLogTest, DistributedFacetSimpleRefinementLongTailTest, SimplePostToolTest, TestFieldCollectionResource, DistanceUnitsTest, ClusterStateUpdateTest, HdfsNNFailoverTest, TestXmlQParser, TestLargeCluster, TestSerializedLuceneMatchVersion, SolrCoreMetricManagerTest, OverseerCollectionConfigSetProcessorTest, SpellCheckCollatorWithCollapseTest, CachingDirectoryFactoryTest, TestHashPartitioner, BooleanFieldTest, TestSubQueryTransformerDistrib, PeerSyncWithIndexFingerprintCachingTest, TestDynamicFieldCollectionResource, TestDocSet, TestCoreBackup, JsonLoaderTest, TestTestInjection, TestNonDefinedSimilarityFactory, TestFreeTextSuggestions, GraphQueryTest, TestStressRecovery, AutoscalingHistoryHandlerTest, RecoveryAfterSoftCommitTest, TestManagedSynonymGraphFilterFactory, ExplicitHLLTest, DistributedQueryComponentOptimizationTest, DirectUpdateHandlerTest, ReturnFieldsTest, OverseerTaskQueueTest, TestManagedStopFilterFactory, TestMaxScoreQueryParser, DistributedTermsComponentTest, HdfsRecoveryZkTest, TestRandomRequestDistribution, HdfsLockFactoryTest, TestSolrQueryResponse, ZkSolrClientTest, TestConfigSetsAPI, TestSQLHandlerNonCloud, TriggerIntegrationTest]
   [junit4] Completed [323/775 (1!)] on J1 in 193.77s, 14 tests, 1 failure <<< FAILURES!

[...truncated 49637 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4424 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4424/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  org.apache.solr.cloud.ReplaceNodeNoTargetTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([E71A15FE6A56B9FE:6F4E2A24C4AAD406]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.ReplaceNodeNoTargetTest.test(ReplaceNodeNoTargetTest.java:92)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost

Error Message:
The operations computed by ComputePlanAction should not be null SolrClientNodeStateProvider.DEBUG{AFTER_ACTION=[compute_plan, null], BEFORE_ACTION=[compute_plan, null]}

Stack Trace:
java.lang.AssertionError: The operations computed by ComputePlanAction should not be null SolrClientNodeStateProvider.DEBUG{AFTER_ACTION=[compute_plan, null], BEFORE_ACTION=[compute_plan, null]}
        at __randomizedtesting.SeedInfo.seed([E71A15FE6A56B9FE:D7DAF47CE22458A2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.autoscaling.ComputePlanActionTest.testNodeWithMultipleReplicasLost(ComputePlanActionTest.java:291)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([E71A15FE6A56B9FE:CCE5C0A5F02EAC2E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTriggerRestoreState(TriggerIntegrationTest.java:368)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTrigger

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([E71A15FE6A56B9FE:E3E4BCEC509C3B64]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTrigger(TriggerIntegrationTest.java:521)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13738 lines...]
   [junit4] Suite: org.apache.solr.cloud.ReplaceNodeNoTargetTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/init-core-data-001
   [junit4]   2> 2322099 WARN  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 2322099 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2322102 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2322102 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2322103 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 6 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001
   [junit4]   2> 2322103 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2322103 INFO  (Thread-4232) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2322103 INFO  (Thread-4232) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2322106 ERROR (Thread-4232) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2322210 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64625
   [junit4]   2> 2322222 INFO  (zkConnectionManagerCallback-4833-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322227 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2322227 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2322227 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2322228 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2322228 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2322228 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2322229 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2322229 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2322229 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2322229 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2322230 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2322230 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2322230 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2322230 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2322230 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2322231 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5b3561c{/solr,null,AVAILABLE}
   [junit4]   2> 2322231 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@13d100c9{/solr,null,AVAILABLE}
   [junit4]   2> 2322232 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3468486{HTTP/1.1,[http/1.1]}{127.0.0.1:64627}
   [junit4]   2> 2322232 INFO  (jetty-launcher-4830-thread-3) [    ] o.e.j.s.Server Started @2327236ms
   [junit4]   2> 2322232 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64627}
   [junit4]   2> 2322232 ERROR (jetty-launcher-4830-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2322232 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@545cb171{/solr,null,AVAILABLE}
   [junit4]   2> 2322232 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-04T04:11:08.164Z
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2322233 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f105e97{/solr,null,AVAILABLE}
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6cfe90a{/solr,null,AVAILABLE}
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7ee174eb{HTTP/1.1,[http/1.1]}{127.0.0.1:64630}
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@27d16493{HTTP/1.1,[http/1.1]}{127.0.0.1:64631}
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.e.j.s.Server Started @2327238ms
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-5) [    ] o.e.j.s.Server Started @2327238ms
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64630}
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64631}
   [junit4]   2> 2322234 ERROR (jetty-launcher-4830-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2322234 ERROR (jetty-launcher-4830-thread-5) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2322234 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-04T04:11:08.166Z
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-04T04:11:08.166Z
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@37ff31f1{HTTP/1.1,[http/1.1]}{127.0.0.1:64628}
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-1) [    ] o.e.j.s.Server Started @2327239ms
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64628}
   [junit4]   2> 2322235 ERROR (jetty-launcher-4830-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2322235 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-04T04:11:08.167Z
   [junit4]   2> 2322236 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@c36f364{/solr,null,AVAILABLE}
   [junit4]   2> 2322238 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@469cb190{HTTP/1.1,[http/1.1]}{127.0.0.1:64629}
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-6) [    ] o.e.j.s.Server Started @2327242ms
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64629}
   [junit4]   2> 2322239 ERROR (jetty-launcher-4830-thread-6) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2322239 INFO  (zkConnectionManagerCallback-4837-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@382926d4{HTTP/1.1,[http/1.1]}{127.0.0.1:64632}
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-04T04:11:08.170Z
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-2) [    ] o.e.j.s.Server Started @2327243ms
   [junit4]   2> 2322239 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64632}
   [junit4]   2> 2322239 ERROR (jetty-launcher-4830-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2322240 INFO  (zkConnectionManagerCallback-4841-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322240 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2322240 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2322240 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2322240 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-04T04:11:08.171Z
   [junit4]   2> 2322240 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2322241 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2322241 INFO  (zkConnectionManagerCallback-4839-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322246 INFO  (zkConnectionManagerCallback-4843-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322247 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2322247 INFO  (zkConnectionManagerCallback-4835-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322248 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2322249 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2322253 INFO  (jetty-launcher-4830-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64625/solr
   [junit4]   2> 2322254 INFO  (jetty-launcher-4830-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64625/solr
   [junit4]   2> 2322257 INFO  (zkConnectionManagerCallback-4845-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322258 INFO  (zkConnectionManagerCallback-4853-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322259 INFO  (jetty-launcher-4830-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64625/solr
   [junit4]   2> 2322259 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2322261 INFO  (zkConnectionManagerCallback-4849-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322262 INFO  (jetty-launcher-4830-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64625/solr
   [junit4]   2> 2322266 INFO  (jetty-launcher-4830-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64625/solr
   [junit4]   2> 2322266 INFO  (zkConnectionManagerCallback-4860-thread-1-processing-n:127.0.0.1:64630_solr) [n:127.0.0.1:64630_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322267 INFO  (zkConnectionManagerCallback-4863-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322269 INFO  (zkConnectionManagerCallback-4857-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322270 INFO  (zkConnectionManagerCallback-4867-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322270 INFO  (jetty-launcher-4830-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64625/solr
   [junit4]   2> 2322271 INFO  (zkConnectionManagerCallback-4869-thread-1-processing-n:127.0.0.1:64628_solr) [n:127.0.0.1:64628_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322274 INFO  (zkConnectionManagerCallback-4875-thread-1-processing-n:127.0.0.1:64631_solr) [n:127.0.0.1:64631_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322274 INFO  (zkConnectionManagerCallback-4873-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322275 INFO  (zkConnectionManagerCallback-4879-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322275 INFO  (zkConnectionManagerCallback-4877-thread-1-processing-n:127.0.0.1:64627_solr) [n:127.0.0.1:64627_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322278 INFO  (zkConnectionManagerCallback-4881-thread-1-processing-n:127.0.0.1:64632_solr) [n:127.0.0.1:64632_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2322514 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2322514 INFO  (jetty-launcher-4830-thread-3) [n:127.0.0.1:64627_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2322514 INFO  (jetty-launcher-4830-thread-5) [n:127.0.0.1:64631_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2322515 INFO  (jetty-launcher-4830-thread-2) [n:127.0.0.1:64632_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2322515 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64629_solr
   [junit4]   2> 2322517 INFO  (jetty-launcher-4830-thread-3) [n:127.0.0.1:64627_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64627_solr
   [junit4]   2> 2322517 INFO  (jetty-launcher-4830-thread-5) [n:127.0.0.1:64631_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64631_solr
   [junit4]   2> 2322517 INFO  (jetty-launcher-4830-thread-1) [n:127.0.0.1:64628_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2322518 INFO  (jetty-launcher-4830-thread-2) [n:127.0.0.1:64632_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64632_solr
   [junit4]   2> 2322518 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.c.Overseer Overseer (id=72138287654109200-127.0.0.1:64629_solr-n_0000000000) starting
   [junit4]   2> 2322519 INFO  (jetty-launcher-4830-thread-4) [n:127.0.0.1:64630_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2322520 INFO  (zkCallback-4874-thread-1-processing-n:127.0.0.1:64631_solr) [n:127.0.0.1:64631_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2322541 INFO  (zkCallback-4868-thread-1-processing-n:127.0.0.1:64628_solr) [n:127.0.0.1:64628_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2322556 INFO  (zkCallback-4874-thread-1-processing-n:127.0.0.1:64631_solr) [n:127.0.0.1:64631_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2322556 INFO  (zkCallback-4876-thread-1-processing-n:127.0.0.1:64627_solr) [n:127.0.0.1:64627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2322558 INFO  (zkCallback-4880-thread-1-processing-n:127.0.0.1:64632_solr) [n:127.0.0.1:64632_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2322560 INFO  (zkCallback-4859-thread-1-processing-n:127.0.0.1:64630_solr) [n:127.0.0.1:64630_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2322563 INFO  (zkCallback-4878-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2322577 INFO  (jetty-launcher-4830-thread-1) [n:127.0.0.1:64628_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64628_solr
   [junit4]   2> 2322585 INFO  (jetty-launcher-4830-thread-4) [n:127.0.0.1:64630_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64630_solr
   [junit4]   2> 2322590 INFO  (zkCallback-4859-thread-1-processing-n:127.0.0.1:64630_solr) [n:127.0.0.1:64630_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2322590 INFO  (zkCallback-4868-thread-1-processing-n:127.0.0.1:64628_solr) [n:127.0.0.1:64628_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2322590 INFO  (zkCallback-4876-thread-1-processing-n:127.0.0.1:64627_solr) [n:127.0.0.1:64627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2322590 INFO  (zkCallback-4878-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2322590 INFO  (zkCallback-4880-thread-1-processing-n:127.0.0.1:64632_solr) [n:127.0.0.1:64632_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2322590 INFO  (zkCallback-4874-thread-1-processing-n:127.0.0.1:64631_solr) [n:127.0.0.1:64631_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2322592 INFO  (zkCallback-4859-thread-1-processing-n:127.0.0.1:64630_solr) [n:127.0.0.1:64630_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2322592 INFO  (zkCallback-4868-thread-1-processing-n:127.0.0.1:64628_solr) [n:127.0.0.1:64628_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2322592 INFO  (zkCallback-4880-thread-1-processing-n:127.0.0.1:64632_solr) [n:127.0.0.1:64632_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2322592 INFO  (zkCallback-4878-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2322592 INFO  (zkCallback-4874-thread-1-processing-n:127.0.0.1:64631_solr) [n:127.0.0.1:64631_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2322593 INFO  (zkCallback-4876-thread-1-processing-n:127.0.0.1:64627_solr) [n:127.0.0.1:64627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2322630 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64629_solr
   [junit4]   2> 2322632 INFO  (zkCallback-4868-thread-1-processing-n:127.0.0.1:64628_solr) [n:127.0.0.1:64628_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2322633 INFO  (zkCallback-4874-thread-1-processing-n:127.0.0.1:64631_solr) [n:127.0.0.1:64631_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2322633 INFO  (zkCallback-4859-thread-1-processing-n:127.0.0.1:64630_solr) [n:127.0.0.1:64630_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2322633 INFO  (zkCallback-4880-thread-1-processing-n:127.0.0.1:64632_solr) [n:127.0.0.1:64632_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2322633 INFO  (zkCallback-4878-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2322634 INFO  (zkCallback-4876-thread-1-processing-n:127.0.0.1:64627_solr) [n:127.0.0.1:64627_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2322659 INFO  (jetty-launcher-4830-thread-3) [n:127.0.0.1:64627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64627.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322664 INFO  (jetty-launcher-4830-thread-5) [n:127.0.0.1:64631_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64631.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322666 INFO  (jetty-launcher-4830-thread-2) [n:127.0.0.1:64632_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64632.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322700 INFO  (jetty-launcher-4830-thread-5) [n:127.0.0.1:64631_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64631.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322700 INFO  (jetty-launcher-4830-thread-5) [n:127.0.0.1:64631_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64631.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322701 INFO  (jetty-launcher-4830-thread-5) [n:127.0.0.1:64631_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001/node5/.
   [junit4]   2> 2322701 INFO  (jetty-launcher-4830-thread-3) [n:127.0.0.1:64627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64627.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322701 INFO  (jetty-launcher-4830-thread-2) [n:127.0.0.1:64632_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64632.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322701 INFO  (jetty-launcher-4830-thread-3) [n:127.0.0.1:64627_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64627.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322701 INFO  (jetty-launcher-4830-thread-2) [n:127.0.0.1:64632_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64632.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322702 INFO  (jetty-launcher-4830-thread-3) [n:127.0.0.1:64627_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001/node3/.
   [junit4]   2> 2322703 INFO  (jetty-launcher-4830-thread-2) [n:127.0.0.1:64632_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001/node2/.
   [junit4]   2> 2322729 INFO  (jetty-launcher-4830-thread-4) [n:127.0.0.1:64630_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64630.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322742 INFO  (jetty-launcher-4830-thread-1) [n:127.0.0.1:64628_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64628.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322802 INFO  (jetty-launcher-4830-thread-4) [n:127.0.0.1:64630_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64630.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322802 INFO  (jetty-launcher-4830-thread-4) [n:127.0.0.1:64630_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64630.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322811 INFO  (jetty-launcher-4830-thread-4) [n:127.0.0.1:64630_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001/node4/.
   [junit4]   2> 2322812 INFO  (jetty-launcher-4830-thread-1) [n:127.0.0.1:64628_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64628.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322812 INFO  (jetty-launcher-4830-thread-1) [n:127.0.0.1:64628_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64628.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322813 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64629.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322813 INFO  (jetty-launcher-4830-thread-1) [n:127.0.0.1:64628_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001/node1/.
   [junit4]   2> 2322823 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64629.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322823 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64629.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@719ed03
   [junit4]   2> 2322824 INFO  (jetty-launcher-4830-thread-6) [n:127.0.0.1:64629_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeNoTargetTest_E71A15FE6A56B9FE-001/tempDir-001/node6/.
   [junit4]   2> 2323115 INFO  (zkConnectionManagerCallback-4895-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323120 INFO  (zkConnectionManagerCallback-4899-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323123 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 2323125 INFO  (SUITE-ReplaceNodeNoTargetTest-seed#[E71A15FE6A56B9FE]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64625/solr ready
   [junit4]   2> 2323207 INFO  (TEST-ReplaceNodeNoTargetTest.test-seed#[E71A15FE6A56B9FE]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2323208 INFO  (zkConnectionManagerCallback-4901-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323210 INFO  (TEST-ReplaceNodeNoTargetTest.test-seed#[E71A15FE6A56B9FE]) [    ] o.a.s.c.ReplaceNodeNoTargetTest total_jettys: 6
   [junit4]   2> 2323223 INFO  (qtp584185162-21938) [n:127.0.0.1:64627_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323225 INFO  (qtp26382918-21951) [n:127.0.0.1:64628_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323229 INFO  (qtp1683485166-21957) [n:127.0.0.1:64629_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 2323235 INFO  (qtp1927719456-21946) [n:127.0.0.1:64630_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=4
   [junit4]   2> 2323240 INFO  (qtp852612644-21960) [n:127.0.0.1:64631_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 2323245 INFO  (qtp994348787-21952) [n:127.0.0.1:64632_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 2323247 INFO  (qtp26382918-21947) [n:127.0.0.1:64628_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=32
   [junit4]   2> 2323249 INFO  (qtp1683485166-21962) [n:127.0.0.1:64629_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=2&collection.configName=conf1&name=replacenodetest_coll_notarget&nrtReplicas=2&action=CREATE&numShards=5&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2323252 INFO  (OverseerThreadFactory-10229-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection replacenodetest_coll_notarget
   [junit4]   2> 2323258 INFO  (qtp584185162-21942) [n:127.0.0.1:64627_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323259 INFO  (qtp26382918-21951) [n:127.0.0.1:64628_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323261 INFO  (qtp1683485166-21963) [n:127.0.0.1:64629_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323262 INFO  (qtp1927719456-21982) [n:127.0.0.1:64630_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323264 INFO  (qtp852612644-21978) [n:127.0.0.1:64631_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323265 INFO  (qtp994348787-21973) [n:127.0.0.1:64632_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2323270 INFO  (OverseerThreadFactory-10229-thread-1-processing-n:127.0.0.1:64629_solr) [n:127.0.0.1:64629_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 1231288033 sessionWrapper.createTime 1231288028928289, this.sessionWrapper.createTime 1231288028928289
   [junit4]   2> 2323386 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64632/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323391 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64631/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323394 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64630/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323397 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64629/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323401 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard3_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64628/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323405 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard3_replica_n10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64627/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323410 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard4",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard4_replica_n12",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64627/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323415 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard4",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard4_replica_n14",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64628/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323420 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard5",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard5_replica_n16",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64629/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323424 INFO  (OverseerStateUpdate-72138287654109200-127.0.0.1:64629_solr-n_0000000000) [n:127.0.0.1:64629_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"replacenodetest_coll_notarget",
   [junit4]   2>   "shard":"shard5",
   [junit4]   2>   "core":"replacenodetest_coll_notarget_shard5_replica_n18",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64630/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2323640 INFO  (qtp994348787-21973) [n:127.0.0.1:64632_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_notarget_shard1_replica_n1&action=CREATE&numShards=5&collection=replacenodetest_coll_notarget&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2323640 INFO  (qtp852612644-21978) [n:127.0.0.1:64631_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_notarget_shard1_replica_n2&action=CREATE&numShards=5&collection=replacenodetest_coll_notarget&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2323640 INFO  (qtp1927719456-21982) [n:127.0.0.1:64630_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf1&newCollection=true&name=replacenodetest_coll_notarget_shard2_replica_n4&action=CREATE&numShards=5&collection=replacenodetest_coll_notarget&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2323640 INFO  (qtp994348787-21973) [n:127.0.0.1:64632_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2323640 INFO  (qtp852612644-21978) [n:127.0.0.1:64631_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2323641 INFO  (qtp1683485166-21963) [n:127.0.0.1:64629_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node9&collection.configName=conf1&newCollection=true&name=replacenodet