[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1667 - Still unstable!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1667 - Still unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1667/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=14974, name=jetty-launcher-2540-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)    2) Thread[id=14978, name=jetty-launcher-2540-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
   1) Thread[id=14974, name=jetty-launcher-2540-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   2) Thread[id=14978, name=jetty-launcher-2540-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
        at __randomizedtesting.SeedInfo.seed([8B983CBE43E57927]:0)


FAILED:  org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory

Error Message:
expected:<5> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<0>
        at __randomizedtesting.SeedInfo.seed([8B983CBE43E57927:E6649843F9AD8620]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:274)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:     1) Thread[id=2532, name=jetty-launcher-538-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)    2) Thread[id=2535, name=jetty-launcher-538-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:
   1) Thread[id=2532, name=jetty-launcher-538-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   2) Thread[id=2535, name=jetty-launcher-538-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
        at __randomizedtesting.SeedInfo.seed([8B983CBE43E57927]:0)




Build Log:
[...truncated 12215 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_8B983CBE43E57927-001/init-core-data-001
   [junit4]   2> 398835 WARN  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3 numCloses=3
   [junit4]   2> 398835 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 398836 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 398836 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 398926 WARN  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   2> 399088 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_8B983CBE43E57927-001/tempDir-001
   [junit4]   2> 399088 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 399088 INFO  (Thread-474) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 399089 INFO  (Thread-474) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 399103 ERROR (Thread-474) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 399189 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:52715
   [junit4]   2> 399191 INFO  (zkConnectionManagerCallback-541-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399217 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T07:27:37+10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 399219 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T07:27:37+10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 399220 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 399222 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 399222 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 399225 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5098fa0a{/solr,null,AVAILABLE}
   [junit4]   2> 399226 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 399226 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 399226 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 399228 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@327d1b73{/solr,null,AVAILABLE}
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@745624e7{HTTP/1.1,[http/1.1]}{127.0.0.1:44031}
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.e.j.s.Server Started @403091ms
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44031}
   [junit4]   2> 399229 ERROR (jetty-launcher-538-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 399229 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T06:33:40.145Z
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@271dc45f{HTTP/1.1,[http/1.1]}{127.0.0.1:50766}
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.e.j.s.Server Started @403092ms
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50766}
   [junit4]   2> 399231 ERROR (jetty-launcher-538-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 399231 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T06:33:40.147Z
   [junit4]   2> 399233 INFO  (zkConnectionManagerCallback-543-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399234 INFO  (zkConnectionManagerCallback-545-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399234 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 399235 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 399247 INFO  (jetty-launcher-538-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52715/solr
   [junit4]   2> 399249 INFO  (zkConnectionManagerCallback-549-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399268 INFO  (zkConnectionManagerCallback-551-thread-1-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399297 INFO  (jetty-launcher-538-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52715/solr
   [junit4]   2> 399299 INFO  (zkConnectionManagerCallback-556-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399303 INFO  (zkConnectionManagerCallback-558-thread-1-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 399325 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 399325 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44031_solr
   [junit4]   2> 399327 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.Overseer Overseer (id=72167303819493380-127.0.0.1:44031_solr-n_0000000000) starting
   [junit4]   2> 399339 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44031_solr
   [junit4]   2> 399374 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 399518 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 399521 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 399524 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50766_solr
   [junit4]   2> 399526 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 399527 INFO  (zkCallback-557-thread-1-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 399551 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 399593 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 399605 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=staff, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 399606 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=staff, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 399730 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 399765 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 399807 INFO  (jetty-launcher-538-thread-1-EventThread) [n:127.0.0.1:50766_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 399810 INFO  (jetty-launcher-538-thread-2-EventThread) [n:127.0.0.1:44031_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 400119 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 400144 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 400201 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44031.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@550fef07
   [junit4]   2> 400202 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50766.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@550fef07
   [junit4]   2> 400210 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44031.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@550fef07
   [junit4]   2> 400211 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44031.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@550fef07
   [junit4]   2> 400211 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50766.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@550fef07
   [junit4]   2> 400211 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50766.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@550fef07
   [junit4]   2> 400212 INFO  (jetty-launcher-538-thread-2) [n:127.0.0.1:44031_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_8B983CBE43E57927-001/tempDir-001/node1/.
   [junit4]   2> 400231 INFO  (jetty-launcher-538-thread-1) [n:127.0.0.1:50766_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_8B983CBE43E57927-001/tempDir-001/node2/.
   [junit4]   2> 400273 INFO  (zkConnectionManagerCallback-563-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 400278 INFO  (zkConnectionManagerCallback-567-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 400284 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 400285 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52715/solr ready
   [junit4]   2> 400316 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 400412 WARN  (qtp374229607-2490) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
   [junit4]   2> id: invalid user name: "bar"
   [junit4]   2>
   [junit4]   2> 400764 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bogusGroup&p=bogusGroup&t=simple-dt&e=1518194021252&s=LlrI5pNM9sYZPqH2uk1hcRWHMn0="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:41 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:41 GMT
   [junit4]   2> 400768 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 400791 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 400823 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 400824 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 400824 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostUsersGroup&p=anyHostUsersGroup&t=simple-dt&e=1518194021710&s=GnJba7fKWka+d60SqAbErKV12tc="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:41 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:41 GMT
   [junit4]   2> 400824 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4] IGNOR/A 0.00s J1 | TestImpersonationWithHadoopAuth.testForwarding
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-9893))
   [junit4]   2> 400844 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 415973 WARN  (jetty-launcher-538-thread-1-SendThread(127.0.0.1:52715)) [n:127.0.0.1:50766_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15745ms for sessionid 0x10063c7cc8f0006
   [junit4]   2> 415974 WARN  (jetty-launcher-538-thread-2-SendThread(127.0.0.1:52715)) [n:127.0.0.1:44031_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15885ms for sessionid 0x10063c7cc8f0008
   [junit4]   2> 415974 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10063c7cc8f0006, likely client has closed socket
   [junit4]   2> 415974 WARN  (jetty-launcher-538-thread-1-SendThread(127.0.0.1:52715)) [n:127.0.0.1:50766_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15802ms for sessionid 0x10063c7cc8f0007
   [junit4]   2> 415974 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10063c7cc8f0008, likely client has closed socket
   [junit4]   2> 415974 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10063c7cc8f0007, likely client has closed socket
   [junit4]   2> 415974 WARN  (jetty-launcher-538-thread-2-SendThread(127.0.0.1:52715)) [n:127.0.0.1:44031_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15570ms for sessionid 0x10063c7cc8f0004
   [junit4]   2> 415975 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10063c7cc8f0004, likely client has closed socket
   [junit4]   2> 415982 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=wrongHost&p=wrongHost&t=simple-dt&e=1518194021762&s=Zm9pyCQ0COgJN2/Z1wIl7xEsBdE="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:41 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:41 GMT
   [junit4]   2> 415982 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 416001 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 416010 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 416010 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 416010 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=simple-dt&e=1518194036925&s=lUBSq0kktuFwyxEUJcl+DHCMS5Q="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:56 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:56 GMT
   [junit4]   2> 416011 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 416036 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 416041 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=simple-dt&e=1518194036954&s=fev0CWcQ5MZl5BQa7FRXzCkikjA="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:56 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:56 GMT
   [junit4]   2> 416041 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 416066 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 416068 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 416068 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 416069 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostAnyUser&p=anyHostAnyUser&t=simple-dt&e=1518194036984&s=DSh9OWCVsNowdtqertaKDy3xqgM="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:56 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:56 GMT
   [junit4]   2> 416069 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 416075 WARN  (zkConnectionManagerCallback-551-thread-1-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e59a6c name: ZooKeeperConnection Watcher:127.0.0.1:52715/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 416076 WARN  (zkConnectionManagerCallback-551-thread-1-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 416079 INFO  (jetty-launcher-538-thread-2-EventThread) [n:127.0.0.1:44031_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 416081 WARN  (zkConnectionManagerCallback-567-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@708b6101 name: ZooKeeperConnection Watcher:127.0.0.1:52715/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 416081 WARN  (zkConnectionManagerCallback-567-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 416088 WARN  (zkConnectionManagerCallback-558-thread-1-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c0fb3be name: ZooKeeperConnection Watcher:127.0.0.1:52715/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 416089 WARN  (zkConnectionManagerCallback-558-thread-1-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 416106 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 416108 INFO  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:449)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:349)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:530)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
   [junit4]   2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 416109 WARN  (qtp374229607-2496) [n:127.0.0.1:44031_solr    ] o.e.j.s.HttpChannel /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication,
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:455)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:349)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:530)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
   [junit4]   2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:449)
   [junit4]   2> ... 31 more
   [junit4]   2> 416115 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4]   2> 416144 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 416161 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noGroups&p=noGroups&t=simple-dt&e=1518194037063&s=ifJ918X1MLkUTOTtkfcFnx+I1Z8="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:57 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:57 GMT
   [junit4]   2> 416161 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 416189 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 416194 WARN  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[8B983CBE43E57927]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noHosts&p=noHosts&t=simple-dt&e=1518194037107&s=De+mMCIYLszFqawaMYHRhGH/e70="; Domain=127.0.0.1; Expires=vie, 09-feb-2018 16:33:57 GMT; HttpOnly". Invalid 'expires' attribute: vie, 09-feb-2018 16:33:57 GMT
   [junit4]   2> 416194 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[8B983CBE43E57927]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 416655 INFO  (jetty-closer-539-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@271dc45f{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 416657 INFO  (jetty-closer-539-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@745624e7{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 416677 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1736115162
   [junit4]   2> 416678 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=213647926
   [junit4]   2> 416683 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 416684 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@854873d: rootName = solr_44031, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@64e11e2
   [junit4]   2> 416687 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 416687 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1d64a9d6: rootName = solr_50766, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@3f11d85e
   [junit4]   2> 416697 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 416697 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@33615380: rootName = solr_44031, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@793d9ffe
   [junit4]   2> 416702 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 416702 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4f5b4a8a: rootName = solr_44031, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@7d020401
   [junit4]   2> 416704 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 416704 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@91c8002: rootName = solr_50766, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@793d9ffe
   [junit4]   2> 416709 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 416709 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@78e34010: rootName = solr_50766, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@7d020401
   [junit4]   2> 417456 INFO  (zkConnectionManagerCallback-551-thread-1-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417462 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:44031_solr
   [junit4]   2> 417463 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 417463 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 417463 INFO  (jetty-closer-539-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72167303819493380-127.0.0.1:44031_solr-n_0000000000) closing
   [junit4]   2> 417464 INFO  (OverseerCollectionConfigSetProcessor-72167303819493380-127.0.0.1:44031_solr-n_0000000000) [n:127.0.0.1:44031_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=72167303819493380-127.0.0.1:44031_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 417466 INFO  (OverseerStateUpdate-72167303819493380-127.0.0.1:44031_solr-n_0000000000) [n:127.0.0.1:44031_solr    ] o.a.s.c.Overseer According to ZK I (id=72167303819493380-127.0.0.1:44031_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 417466 INFO  (OverseerStateUpdate-72167303819493380-127.0.0.1:44031_solr-n_0000000000) [n:127.0.0.1:44031_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:44031_solr
   [junit4]   2> 417466 WARN  (OverseerAutoScalingTriggerThread-72167303819493380-127.0.0.1:44031_solr-n_0000000000) [n:127.0.0.1:44031_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 417469 INFO  (zkCallback-550-thread-8-processing-n:127.0.0.1:44031_solr) [n:127.0.0.1:44031_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 417472 ERROR (Thread[Thread-491,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:44031_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 417551 INFO  (jetty-closer-539-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@327d1b73{/solr,null,UNAVAILABLE}
   [junit4]   2> 417564 INFO  (jetty-closer-539-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 417653 INFO  (zkConnectionManagerCallback-558-thread-1-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 417654 INFO  (zkCallback-557-thread-7-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 417654 INFO  (zkCallback-557-thread-8-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50766_solr
   [junit4]   2> 417655 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50766_solr
   [junit4]   2> 417656 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 417656 INFO  (jetty-closer-539-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 417821 INFO  (jetty-launcher-538-thread-1-EventThread) [n:127.0.0.1:50766_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 419159 WARN  (zkCallback-557-thread-7-processing-n:127.0.0.1:50766_solr) [n:127.0.0.1:50766_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 419160 ERROR (Thread[Thread-493,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:50766_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 419175 INFO  (jetty-closer-539-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5098fa0a{/solr,null,UNAVAILABLE}
   [junit4]   2> 419175 INFO  (jetty-closer-539-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 419177 ERROR (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 419177 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52715 52715
   [junit4]   2> 424318 INFO  (Thread-474) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52715 52715
   [junit4]   2> 424320 WARN  (Thread-474) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/aliases.json
   [junit4]   2> 3 /solr/clusterprops.json
   [junit4]   2> 2 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 3 /solr/clusterstate.json
   [junit4]   2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 4 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 3 /solr/live_nodes
   [junit4]   2> 3 /solr/collections
   [junit4]   2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2>
   [junit4]   2> 424320 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Feb 09, 2018 6:34:05 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Feb 09, 2018 6:34:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:
   [junit4]   2>    1) Thread[id=2532, name=jetty-launcher-538-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   [junit4]   2>    2) Thread[id=2535, name=jetty-launcher-538-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   [junit4]   2> Feb 09, 2018 6:34:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=2532, name=jetty-launcher-538-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>    2) Thread[id=2535, name=jetty-launcher-538-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2> 444519 ERROR (jetty-launcher-538-thread-2-EventThread) [n:127.0.0.1:44031_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   [junit4]   2> 444519 ERROR (jetty-launcher-538-thread-1-EventThread) [n:127.0.0.1:50766_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   [junit4]   2> Feb 09, 2018 6:34:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=76, maxMBSortInHeap=7.391666528781155, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@437560cf), locale=es-US, timezone=Africa/Accra
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=70533208,total=259493888
   [junit4]   2> NOTE: All tests run in this JVM: [SegmentsInfoRequestHandlerTest, BlockJoinFacetRandomTest, TestUseDocValuesAsStored2, TestNumericTerms64, BooleanFieldTest, BJQParserTest, SolrCoreCheckLockOnStartupTest, TestManagedStopFilterFactory, DefaultValueUpdateProcessorTest, TestAnalyzedSuggestions, ReplicationFactorTest, DeleteStatusTest, OverseerRolesTest, PreAnalyzedFieldTest, TestCoreDiscovery, TestSortableTextField, TestSha256AuthenticationProvider, TestGraphMLResponseWriter, PeerSyncWithIndexFingerprintCachingTest, TestCloudSearcherWarming, TestNRTOpen, SpellCheckComponentTest, TestWriterPerf, NodeLostTriggerTest, TestTlogReplica, DeleteLastCustomShardedReplicaTest, CleanupOldIndexTest, TestSolr4Spatial2, MetricTriggerTest, SortSpecParsingTest, DistributedQueueTest, URLClassifyProcessorTest, TestComponentsName, TestImpersonationWithHadoopAuth]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=8B983CBE43E57927 -Dtests.slow=true -Dtests.locale=es-US -Dtests.timezone=Africa/Accra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestImpersonationWithHadoopAuth (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:
   [junit4]    >    1) Thread[id=2532, name=jetty-launcher-538-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   [junit4]    >    2) Thread[id=2535, name=jetty-launcher-538-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([8B983CBE43E57927]:0)
   [junit4] Completed [75/775 (1!)] on J1 in 45.70s, 10 tests, 1 error, 1 skipped <<< FAILURES!

[...truncated 690 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8B983CBE43E57927-001/init-core-data-001
   [junit4]   2> 1581525 WARN  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1581526 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1581527 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1581527 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1581567 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8B983CBE43E57927-001/tempDir-001
   [junit4]   2> 1581567 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1581567 INFO  (Thread-4936) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1581567 INFO  (Thread-4936) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1581568 ERROR (Thread-4936) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1581667 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8B983CBE43E57927]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44510
   [junit4]   2> 1581669 INFO  (zkConnectionManagerCallback-2543-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581675 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T07:27:37+10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1581675 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T07:27:37+10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1581676 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@97d7a36{/solr,null,AVAILABLE}
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1af982e{/solr,null,AVAILABLE}
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@53cff17{HTTP/1.1,[http/1.1]}{127.0.0.1:63016}
   [junit4]   2> 1581677 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@8a4192c{HTTP/1.1,[http/1.1]}{127.0.0.1:45983}
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-2) [    ] o.e.j.s.Server Started @1586975ms
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-1) [    ] o.e.j.s.Server Started @1586975ms
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=63016}
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45983}
   [junit4]   2> 1581678 ERROR (jetty-launcher-2540-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1581678 ERROR (jetty-launcher-2540-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T06:53:22.594Z
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1581678 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T06:53:22.594Z
   [junit4]   2> 1581680 INFO  (zkConnectionManagerCallback-2545-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581680 INFO  (zkConnectionManagerCallback-2547-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581681 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1581681 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1581692 INFO  (jetty-launcher-2540-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44510/solr
   [junit4]   2> 1581693 INFO  (zkConnectionManagerCallback-2551-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581694 INFO  (jetty-launcher-2540-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44510/solr
   [junit4]   2> 1581695 INFO  (zkConnectionManagerCallback-2555-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581698 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10063d9dd380004, likely client has closed socket
   [junit4]   2> 1581698 INFO  (zkConnectionManagerCallback-2557-thread-1-processing-n:127.0.0.1:45983_solr) [n:127.0.0.1:45983_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581700 INFO  (zkConnectionManagerCallback-2559-thread-1-processing-n:127.0.0.1:63016_solr) [n:127.0.0.1:63016_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581781 INFO  (jetty-launcher-2540-thread-2) [n:127.0.0.1:63016_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1581782 INFO  (jetty-launcher-2540-thread-2) [n:127.0.0.1:63016_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:63016_solr
   [junit4]   2> 1581783 INFO  (jetty-launcher-2540-thread-2) [n:127.0.0.1:63016_solr    ] o.a.s.c.Overseer Overseer (id=72167381408415750-127.0.0.1:63016_solr-n_0000000000) starting
   [junit4]   2> 1581792 INFO  (jetty-launcher-2540-thread-2) [n:127.0.0.1:63016_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63016_solr
   [junit4]   2> 1581803 INFO  (jetty-launcher-2540-thread-1) [n:127.0.0.1:45983_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1581808 INFO  (jetty-launcher-2540-thread-1) [n:127.0.0.1:45983_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45983_solr
   [junit4]   2> 1581849 INFO  (OverseerStateUpdate-72167381408415750-127.0.0.1:63016_solr-n_0000000000) [n:127.0.0.1:63016_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1581860 INFO  (zkCallback-2556-thread-1-processing-n:127.0.0.1:45983_solr) [n:127.0.0.1:45983_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1581899 INFO  (jetty-launcher-2540-thread-1) [n:127.0.0.1:45983_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 1581905 WARN  (jetty-launcher-2540-thread-1) [n:127.0.0.1:45983_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 1581907 INFO  (jetty-launcher-2540-thread-1) [n:127.0.0.1:45983_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.

[...truncated too long message...]

t_collection/state.json
   [junit4]   2> 13 /solr/clusterstate.json
   [junit4]   2> 2 /solr/collections/AutoscalingHistoryHandlerTest_collection/leader_elect/shard1/election/72167545512722439-core_node6-n_0000000000
   [junit4]   2> 2 /solr/collections/.system/leader_elect/shard1/election/72167545512722441-core_node3-n_0000000000
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2> 2 /solr/overseer_elect/election/72167545512722440-127.0.0.1:39464_solr-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 13 /solr/live_nodes
   [junit4]   2> 13 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/autoscaling/events/AutoscalingHistoryHandlerTest_node_lost_trigger
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_8B983CBE43E57927-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {action_s=FST50, event.source_s=PostingsFormat(name=Memory), type=FST50, before.actions_ss=PostingsFormat(name=Memory), event.property._enqueue_time__ss=FST50, after.actions_ss=BlockTreeOrds(blocksize=128), stage_s=FST50, event.id_s=PostingsFormat(name=Asserting), event.property.eventTimes_ss=FST50, source_s=PostingsFormat(name=Asserting), event.type_s=FST50, responses_ts=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=Asserting), collections_ss=FST50, operations.params_ts=FST50, event.property.nodeNames_ss=FST50}, docValues:{_version_=DocValuesFormat(name=Memory), event.time_l=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=453, maxMBSortInHeap=5.190096188777827, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4fad7fb6), locale=pt-BR, timezone=Asia/Riyadh
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=255616976,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [SegmentsInfoRequestHandlerTest, BlockJoinFacetRandomTest, TestUseDocValuesAsStored2, TestNumericTerms64, BooleanFieldTest, BJQParserTest, SolrCoreCheckLockOnStartupTest, TestManagedStopFilterFactory, DefaultValueUpdateProcessorTest, TestAnalyzedSuggestions, ReplicationFactorTest, DeleteStatusTest, OverseerRolesTest, PreAnalyzedFieldTest, TestCoreDiscovery, TestSortableTextField, TestSha256AuthenticationProvider, TestGraphMLResponseWriter, PeerSyncWithIndexFingerprintCachingTest, TestCloudSearcherWarming, TestNRTOpen, SpellCheckComponentTest, TestWriterPerf, NodeLostTriggerTest, TestTlogReplica, DeleteLastCustomShardedReplicaTest, CleanupOldIndexTest, TestSolr4Spatial2, MetricTriggerTest, SortSpecParsingTest, DistributedQueueTest, URLClassifyProcessorTest, TestComponentsName, TestImpersonationWithHadoopAuth, TestLuceneIndexBackCompat, ReplicaListTransformerTest, DeleteReplicaTest, TestExecutePlanAction, TestSolrQueryResponse, TestDocSet, TestLeaderElectionWithEmptyReplica, FileBasedSpellCheckerTest, TestManagedSynonymGraphFilterFactory, BasicFunctionalityTest, DistributedSpellCheckComponentTest, TestCollationFieldDocValues, TestCollapseQParserPlugin, SuggestComponentTest, DistributedExpandComponentTest, TestSubQueryTransformerDistrib, TestFieldSortValues, DistributedFacetSimpleRefinementLongTailTest, SimpleFacetsTest, TestSQLHandlerNonCloud, MoveReplicaHDFSTest, HdfsWriteToMultipleCollectionsTest, SimpleCollectionCreateDeleteTest, TestLMDirichletSimilarityFactory, TestWordDelimiterFilterFactory, TestReloadDeadlock, SpellPossibilityIteratorTest, UpdateParamsTest, SolrShardReporterTest, DistributedTermsComponentTest, TestAtomicUpdateErrorCases, TestSchemaSimilarityResource, SolrCmdDistributorTest, TestFieldCache, SubstringBytesRefFilterTest, AutoAddReplicasPlanActionTest, AnalyticsQueryTest, ExitableDirectoryReaderTest, TolerantUpdateProcessorTest, DocValuesMissingTest, TestExactStatsCache, ResponseLogComponentTest, TestRandomCollapseQParserPlugin, TestLargeCluster, LoggingHandlerTest, SuggesterWFSTTest, TestDynamicLoading, MissingSegmentRecoveryTest, TestDocBasedVersionConstraints, TestDistributedGrouping, TestLegacyTerms, EchoParamsTest, PeerSyncTest, JavabinLoaderTest, CdcrReplicationHandlerTest, ClusterStateUpdateTest, SyncSliceTest, OverseerCollectionConfigSetProcessorTest, TestEmbeddedSolrServerConstructors, CollectionsAPIAsyncDistributedZkTest, TestLegacyFieldReuse, CdcrUpdateLogTest, TestCoreBackup, TestPushWriter, TestSolrFieldCacheBean, JSONWriterTest, TestCloudPseudoReturnFields, TestSolrConfigHandler, MergeStrategyTest, TestConfigOverlay, ShowFileRequestHandlerTest, TestMiniSolrCloudClusterSSL, TestBadConfig, CoreAdminRequestStatusTest, DistributedIntervalFacetingTest, CircularListTest, ChaosMonkeyNothingIsSafeTest, SolrSlf4jReporterTest, BufferStoreTest, HdfsCollectionsAPIDistributedZkTest, TestStressReorder, TestCloudJSONFacetJoinDomain, NotRequiredUniqueKeyTest, TestJsonFacets, SolrCoreMetricManagerTest, TestDeleteCollectionOnDownNodes, TestRandomDVFaceting, QueryResultKeyTest, TestNamedUpdateProcessors, BadIndexSchemaTest, ReturnFieldsTest, SystemLogListenerTest, OutOfBoxZkACLAndCredentialsProvidersTest, CursorMarkTest, TestFastWriter, TestNodeLostTrigger, TestNonDefinedSimilarityFactory, BlockCacheTest, IndexSchemaTest, SpellCheckCollatorTest, DistributedFacetPivotWhiteBoxTest, TestSizeLimitedDistributedMap, ResponseHeaderTest, TestClusterStateMutator, TestReRankQParserPlugin, TestSchemaManager, TestBinaryResponseWriter, TermVectorComponentTest, SortByFunctionTest, TestFastLRUCache, TestManagedSchemaThreadSafety, CdcrVersionReplicationTest, AsyncCallRequestStatusResponseTest, DocValuesTest, CSVRequestHandlerTest, TestComputePlanAction, TestSolrDeletionPolicy2, TestConfigSetsAPIZkFailure, SystemInfoHandlerTest, SchemaApiFailureTest, TestStressRecovery, UpdateRequestProcessorFactoryTest, TestSolrCloudWithSecureImpersonation, HdfsSyncSliceTest, CurrencyFieldTypeTest, DocValuesMultiTest, HdfsNNFailoverTest, TestPivotHelperCode, HealthCheckHandlerTest, TestRandomFaceting, TestPKIAuthenticationPlugin, TestMaxTokenLenTokenizer, TestPayloadScoreQParserPlugin, ConcurrentCreateRoutedAliasTest, HdfsAutoAddReplicasIntegrationTest, RegexBoostProcessorTest, TestRTimerTree, TestCorePropertiesReload, BaseCdcrDistributedZkTest, TestNumericTokenStream, TestValueSourceCache, ChaosMonkeySafeLeaderWithPullReplicasTest, DistributedQueryComponentOptimizationTest, TestInitQParser, PrimUtilsTest, TestLMJelinekMercerSimilarityFactory, TestLRUCache, JvmMetricsTest, TestXmlQParser, TestBlobHandler, TestInfoStreamLogging, ZkSolrClientTest, SimplePostToolTest, SolrGangliaReporterTest, TestDocumentBuilder, TestRandomRequestDistribution, ComputePlanActionTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, LeaderElectionTest, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, TestJoin, TestReload, TestStressVersions, TestRangeQuery, TestSolr4Spatial, StatsComponentTest, TestGroupingSearch, ConvertedLegacyTest, TestSort, TestLazyCores, DirectUpdateHandlerTest, SoftAutoCommitTest, DebugComponentTest, IndexBasedSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, SolrPluginUtilsTest, JsonLoaderTest, HighlighterConfigTest, TestQuerySenderListener, TestSolrIndexConfig, BadComponentTest, TestBinaryField, TestSearchPerf, MinimalSchemaTest, OutputWriterTest, ExternalFileFieldSortTest, DirectSolrConnectionTest, TestPhraseSuggestions, TestCodecSupport, TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, SpellingQueryConverterTest, ClusterStateTest, ZkNodePropsTest, FileUtilsTest, TestRTGBase, CursorPagingTest, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestTolerantSearch, ActionThrottleTest, AliasIntegrationTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DistribCursorPagingTest, DocValuesNotIndexedTest, ForceLeaderTest, LegacyCloudClusterPropTest, OverseerTaskQueueTest, PeerSyncReplicationTest, SSLMigrationTest, SaslZkACLProviderTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestClusterProperties, TestConfigSetsAPI, TestDistribDocBasedVersion, TestLeaderInitiatedRecoveryThread, TestPrepRecovery, TestPullReplicaErrorHandling, TestSolrCloudWithDelegationTokens, TestStressInPlaceUpdates, ZkShardTermsTest, CollectionReloadTest, CollectionTooManyReplicasTest, CollectionsAPIDistributedZkTest, TestLocalFSCloudBackupRestore, TestReplicaProperties, AutoScalingHandlerTest, ExecutePlanActionTest, TestClusterStateProvider, TestDistribStateManager, TestGenericDistributedQueue, TestSimDistributedQueue, TestTriggerIntegration, CdcrBootstrapTest, CdcrReplicationDistributedZkTest, HdfsBasicDistributedZk2Test, HdfsChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, HdfsRecoverLeaseTest, HdfsRecoveryZkTest, HdfsThreadLeakTest, HdfsTlogReplayBufferedWhileIndexingTest, HdfsUnloadDistributedZkTest, ZkStateReaderTest, RuleEngineTest, BlobRepositoryCloudTest, DirectoryFactoryTest, TestConfigSetImmutable, TestImplicitCoreProperties, TestInitParams, TestShardHandlerFactory, TestSolrCloudSnapshots, RequestLoggingTest, SearchHandlerTest, TestConfigReload, TestReplicationHandlerBackup, TestRestoreCore, TestSolrConfigHandlerCloud, V2ApiIntegrationTest, V2StandaloneTest, AutoscalingHistoryHandlerTest]
   [junit4] Completed [646/775 (3!)] on J1 in 48.76s, 1 test, 1 failure <<< FAILURES!

[...truncated 48570 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1668 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1668/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger

Error Message:
{srt=[CapturedEvent{timestamp=1518198209427000000, stage=AFTER_ACTION, actionName='compute', event={   "id":"6171c33c8d44fTdd76bgupy4z9rx7h8ho6o06s8",   "source":"search_rate_trigger",   "eventTime":1714259755586639,   "eventType":"SEARCHRATE",   "properties":{     "node":{       "127.0.0.1:54977_solr":38.26277198464036,       "127.0.0.1:58815_solr":33.431061693549694},     "__start__":1,     "replica":[       "{\"core_node3\":{\n    \"base_url\":\"http://127.0.0.1:54977/solr\",\n    \"node_name\":\"127.0.0.1:54977_solr\",\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"rate\":38.26277198464036,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",       "{\"core_node4\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"leader\":\"true\",\n    \"rate\":33.431061693549694,\n    \"base_url\":\"http://127.0.0.1:58815/solr\",\n    \"node_name\":\"127.0.0.1:58815_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],     "collection":{"collection1":71.69383367819006},     "shard":{"collection1":{"shard1":71.69383367819006}},     "_enqueue_time_":1714279868394836,     "requestedOps":[       "Op{action=ADDREPLICA, hints={COLL_SHARD={\n  \"first\":\"collection1\",\n  \"second\":\"shard1\"}}}",       "Op{action=ADDREPLICA, hints={COLL_SHARD={\n  \"first\":\"collection1\",\n  \"second\":\"shard1\"}}}",       "Op{action=ADDREPLICA, hints={COLL_SHARD={\n  \"first\":\"collection1\",\n  \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:42813_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:42813_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:57054_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute], source=search_rate_trigger, properties.AFTER_ACTION=[compute]}, config={   "trigger":"search_rate_trigger",   "stage":[     "FAILED",     "SUCCEEDED"],   "afterAction":[     "compute",     "execute",     "test"],   "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",   "beforeAction":[]}, message='null'}]} expected:<4> but was:<1>

Stack Trace:
java.lang.AssertionError: {srt=[CapturedEvent{timestamp=1518198209427000000, stage=AFTER_ACTION, actionName='compute', event={
  "id":"6171c33c8d44fTdd76bgupy4z9rx7h8ho6o06s8",
  "source":"search_rate_trigger",
  "eventTime":1714259755586639,
  "eventType":"SEARCHRATE",
  "properties":{
    "node":{
      "127.0.0.1:54977_solr":38.26277198464036,
      "127.0.0.1:58815_solr":33.431061693549694},
    "__start__":1,
    "replica":[
      "{\"core_node3\":{\n    \"base_url\":\"http://127.0.0.1:54977/solr\",\n    \"node_name\":\"127.0.0.1:54977_solr\",\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"rate\":38.26277198464036,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
      "{\"core_node4\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"leader\":\"true\",\n    \"rate\":33.431061693549694,\n    \"base_url\":\"http://127.0.0.1:58815/solr\",\n    \"node_name\":\"127.0.0.1:58815_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
    "collection":{"collection1":71.69383367819006},
    "shard":{"collection1":{"shard1":71.69383367819006}},
    "_enqueue_time_":1714279868394836,
    "requestedOps":[
      "Op{action=ADDREPLICA, hints={COLL_SHARD={\n  \"first\":\"collection1\",\n  \"second\":\"shard1\"}}}",
      "Op{action=ADDREPLICA, hints={COLL_SHARD={\n  \"first\":\"collection1\",\n  \"second\":\"shard1\"}}}",
      "Op{action=ADDREPLICA, hints={COLL_SHARD={\n  \"first\":\"collection1\",\n  \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:42813_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:42813_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:57054_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute], source=search_rate_trigger, properties.AFTER_ACTION=[compute]}, config={
  "trigger":"search_rate_trigger",
  "stage":[
    "FAILED",
    "SUCCEEDED"],
  "afterAction":[
    "compute",
    "execute",
    "test"],
  "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",
  "beforeAction":[]}, message='null'}]} expected:<4> but was:<1>
        at __randomizedtesting.SeedInfo.seed([71CDD7A29492139B:CBC1E02DCB7AC5D4]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1565)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12527 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/init-core-data-001
   [junit4]   2> 1220755 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1220758 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1220758 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1220759 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/tempDir-001
   [junit4]   2> 1220759 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1220760 INFO  (Thread-2911) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1220760 INFO  (Thread-2911) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1220762 ERROR (Thread-2911) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1220860 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:47432
   [junit4]   2> 1220863 INFO  (zkConnectionManagerCallback-2457-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1220868 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:27:37+09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1220868 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:27:37+09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1220891 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1220891 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1220891 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1220891 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1220891 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1220891 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1220892 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7568b1d8{/solr,null,AVAILABLE}
   [junit4]   2> 1220892 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1bd9b33{/solr,null,AVAILABLE}
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5499e47d{HTTP/1.1,[http/1.1]}{127.0.0.1:37037}
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.e.j.s.Server Started @1224497ms
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37037}
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3c84ff41{HTTP/1.1,[http/1.1]}{127.0.0.1:39076}
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.e.j.s.Server Started @1224497ms
   [junit4]   2> 1220893 ERROR (jetty-launcher-2454-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39076}
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1220893 ERROR (jetty-launcher-2454-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T17:41:38.722Z
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1220893 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T17:41:38.722Z
   [junit4]   2> 1220896 INFO  (zkConnectionManagerCallback-2459-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1220896 INFO  (zkConnectionManagerCallback-2461-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1220897 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1220897 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1220912 INFO  (jetty-launcher-2454-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47432/solr
   [junit4]   2> 1220915 INFO  (jetty-launcher-2454-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47432/solr
   [junit4]   2> 1220918 INFO  (zkConnectionManagerCallback-2465-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1220943 INFO  (zkConnectionManagerCallback-2471-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1220946 INFO  (zkConnectionManagerCallback-2469-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1220951 INFO  (zkConnectionManagerCallback-2473-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1221118 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1221118 INFO  (jetty-launcher-2454-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1221119 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37037_solr
   [junit4]   2> 1221120 INFO  (jetty-launcher-2454-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39076_solr
   [junit4]   2> 1221121 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.Overseer Overseer (id=72169933809778692-127.0.0.1:37037_solr-n_0000000000) starting
   [junit4]   2> 1221122 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1221123 INFO  (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1221131 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37037_solr
   [junit4]   2> 1221132 INFO  (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1221146 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1221147 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1221147 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1221148 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:37037_solr, 127.0.0.1:39076_solr]
   [junit4]   2> 1221148 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1221148 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1221151 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1221152 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1221153 DEBUG (ScheduledTrigger-3414-thread-1) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1221241 INFO  (jetty-launcher-2454-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39076.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221255 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37037.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221258 INFO  (jetty-launcher-2454-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39076.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221258 INFO  (jetty-launcher-2454-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39076.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221259 INFO  (jetty-launcher-2454-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/tempDir-001/node1/.
   [junit4]   2> 1221261 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37037.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221261 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37037.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221262 INFO  (jetty-launcher-2454-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/tempDir-001/node2/.
   [junit4]   2> 1221295 INFO  (zkConnectionManagerCallback-2479-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1221299 INFO  (zkConnectionManagerCallback-2483-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1221301 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1221303 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:47432/solr ready
   [junit4]   2> 1221327 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 1221327 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 1221328 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1221328 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1221331 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1221332 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1221332 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1221338 INFO  (qtp160109341-10208) [n:127.0.0.1:37037_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1221340 INFO  (qtp215013857-10217) [n:127.0.0.1:39076_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1221340 DEBUG (qtp215013857-10209) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1221341 INFO  (qtp215013857-10209) [n:127.0.0.1:39076_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 1221342 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1221343 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:37037_solr, 127.0.0.1:39076_solr]
   [junit4]   2> 1221343 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1221343 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1221346 INFO  (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1221347 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:27:37+09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1221348 DEBUG (ScheduledTrigger-3414-thread-1) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1221348 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1221348 DEBUG (ScheduledTrigger-3414-thread-1) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1221348 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1221348 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1221348 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1221349 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@13911af4{/solr,null,AVAILABLE}
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6815d345{HTTP/1.1,[http/1.1]}{127.0.0.1:42044}
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.e.j.s.Server Started @1224954ms
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42044}
   [junit4]   2> 1221349 ERROR (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1221349 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T17:41:39.178Z
   [junit4]   2> 1221351 INFO  (zkConnectionManagerCallback-2485-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1221352 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1221364 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47432/solr
   [junit4]   2> 1221366 INFO  (zkConnectionManagerCallback-2489-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1221370 INFO  (zkConnectionManagerCallback-2491-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1221381 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1221385 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1221388 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42044_solr
   [junit4]   2> 1221393 INFO  (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1221395 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1221396 INFO  (zkCallback-2482-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1221396 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1221544 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42044.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221554 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42044.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221554 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42044.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1221555 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [n:127.0.0.1:42044_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/tempDir-001/node3/.
   [junit4]   2> 1222360 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1222360 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1222360 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:42044_solr at time 1714170482155765
   [junit4]   2> 1223362 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1223362 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1223363 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:42044_solr] added at times [1714170482155765], now=1714171484772028
   [junit4]   2> 1223363 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"617076aac60f5Tdd76bgupy4z9rx7h8ho6o06rc",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1714170482155765,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714170482155765],
   [junit4]   2>     "nodeNames":["127.0.0.1:42044_solr"]}}
   [junit4]   2> 1223363 DEBUG (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger]
   [junit4]   2> 1223364 INFO  (ScheduledTrigger-3414-thread-2) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1223365 DEBUG (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"617076aac60f5Tdd76bgupy4z9rx7h8ho6o06rc",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":1714170482155765,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714170482155765],
   [junit4]   2>     "_enqueue_time_":1714171485589790,
   [junit4]   2>     "nodeNames":["127.0.0.1:42044_solr"]}}
   [junit4]   2> 1223367 DEBUG (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 5000ms
   [junit4]   2> 1223367 DEBUG (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=617076aac60f5Tdd76bgupy4z9rx7h8ho6o06rc
   [junit4]   2> 1223369 INFO  (qtp160109341-10213) [n:127.0.0.1:37037_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1223371 INFO  (qtp215013857-10217) [n:127.0.0.1:39076_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1223373 INFO  (qtp1612969104-10270) [n:127.0.0.1:42044_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1223374 DEBUG (qtp160109341-10220) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1223375 INFO  (qtp160109341-10220) [n:127.0.0.1:37037_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 1223375 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1223376 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:37037_solr, 127.0.0.1:42044_solr, 127.0.0.1:39076_solr]
   [junit4]   2> 1223376 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 1223376 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1223377 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1223377 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1225382 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[71CDD7A29492139B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4]   2> 1225408 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 1225409 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
   [junit4]   2> 1225414 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1225414 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1225415 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1225416 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1225416 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1225417 INFO  (qtp160109341-10221) [n:127.0.0.1:37037_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1225428 INFO  (qtp160109341-10221) [n:127.0.0.1:37037_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 1225430 INFO  (qtp160109341-10221) [n:127.0.0.1:37037_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1225431 INFO  (qtp215013857-10211) [n:127.0.0.1:39076_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1225432 INFO  (qtp1612969104-10272) [n:127.0.0.1:42044_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1225432 DEBUG (qtp160109341-10220) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1225433 INFO  (qtp160109341-10220) [n:127.0.0.1:37037_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 1225434 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1225434 INFO  (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 1225434 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:37037_solr, 127.0.0.1:42044_solr, 127.0.0.1:39076_solr]
   [junit4]   2> 1225434 DEBUG (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEQ instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 1225434 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1225437 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1225438 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1225438 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:27:37+09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1225438 DEBUG (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1225439 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1225439 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1225439 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1225439 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f04d222{/solr,null,AVAILABLE}
   [junit4]   2> 1225440 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@61ee6bb5{HTTP/1.1,[http/1.1]}{127.0.0.1:49814}
   [junit4]   2> 1225440 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.Server Started @1229046ms
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49814}
   [junit4]   2> 1225440 ERROR (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1225440 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T17:41:43.269Z
   [junit4]   2> 1225443 INFO  (zkConnectionManagerCallback-2495-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1225443 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1225450 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47432/solr
   [junit4]   2> 1225453 INFO  (zkConnectionManagerCallback-2499-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1225457 INFO  (zkConnectionManagerCallback-2501-thread-1-processing-n:127.0.0.1:49814_solr) [n:127.0.0.1:49814_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1225468 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1225472 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1225476 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49814_solr
   [junit4]   2> 1225478 INFO  (zkCallback-2482-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1225478 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1225479 INFO  (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1225479 INFO  (zkCallback-2500-thread-1-processing-n:127.0.0.1:49814_solr) [n:127.0.0.1:49814_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1225480 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1225657 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49814.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1225664 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49814.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1225664 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49814.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1225665 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [n:127.0.0.1:49814_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/tempDir-001/node4/.
   [junit4]   2> 1226441 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1226441 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1226441 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:49814_solr at time 1714174564793407
   [junit4]   2> 1227424 INFO  (OverseerCollectionConfigSetProcessor-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1227444 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1227444 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1227444 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEQ firing registered processor for nodes: [127.0.0.1:49814_solr] added at times [1714174564793407], now=1714175567534620
   [junit4]   2> 1227444 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"617085e047c3fTdd76bgupy4z9rx7h8ho6o06re",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1714174564793407,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714174564793407],
   [junit4]   2>     "nodeNames":["127.0.0.1:49814_solr"]}}
   [junit4]   2> 1227444 DEBUG (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerEQ]
   [junit4]   2> 1227446 INFO  (ScheduledTrigger-3414-thread-4) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1227446 DEBUG (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"617085e047c3fTdd76bgupy4z9rx7h8ho6o06re",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1714174564793407,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714174564793407],
   [junit4]   2>     "_enqueue_time_":1714175567744088,
   [junit4]   2>     "nodeNames":["127.0.0.1:49814_solr"]}}
   [junit4]   2> 1227447 INFO  (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.TriggerIntegrationTest -- event: {
   [junit4]   2>   "id":"617085e047c3fTdd76bgupy4z9rx7h8ho6o06re",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1714174564793407,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714174564793407],
   [junit4]   2>     "_enqueue_time_":1714175567744088,
   [junit4]   2>     "nodeNames":["127.0.0.1:49814_solr"]}}
   [junit4]   2> 1227448 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5499e47d{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1227449 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1473464151
   [junit4]   2> 1227450 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1227450 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4f3cd07d: rootName = solr_37037, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2a411c9a
   [junit4]   2> 1227458 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1227458 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6d5b25c8: rootName = solr_37037, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@26224cdc
   [junit4]   2> 1227461 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1227461 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@34bc7cf7: rootName = solr_37037, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1378bd3a
   [junit4]   2> 1227463 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:37037_solr
   [junit4]   2> 1227465 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1227465 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1227466 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1227467 INFO  (zkCallback-2470-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1227467 INFO  (zkCallback-2500-thread-1-processing-n:127.0.0.1:49814_solr) [n:127.0.0.1:49814_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1227468 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1227468 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.Overseer Overseer (id=72169933809778692-127.0.0.1:37037_solr-n_0000000000) closing
   [junit4]   2> 1227468 INFO  (OverseerStateUpdate-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37037_solr
   [junit4]   2> 1227469 INFO  (zkCallback-2482-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1227469 WARN  (OverseerAutoScalingTriggerThread-72169933809778692-127.0.0.1:37037_solr-n_0000000000) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1227469 DEBUG (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1227471 WARN  (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_added_triggerEQ has been closed.
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.dequeue(ScheduledTriggers.java:525)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1227474 DEBUG (AutoscalingActionExecutor-3415-thread-1-processing-n:127.0.0.1:37037_solr) [n:127.0.0.1:37037_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 27 ms for event id=617085e047c3fTdd76bgupy4z9rx7h8ho6o06re
   [junit4]   2> 1227474 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7568b1d8{/solr,null,UNAVAILABLE}
   [junit4]   2> 1227480 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1227481 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39076_solr
   [junit4]   2> 1227489 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.Overseer Overseer (id=72169933809778694-127.0.0.1:39076_solr-n_0000000001) starting
   [junit4]   2> 1227499 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1227500 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1227500 INFO  (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 1227501 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:39076_solr]
   [junit4]   2> 1227501 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEQ instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 1227502 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:39076_solr]
   [junit4]   2> 1227502 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 1227502 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1227505 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 1227506 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1227506 DEBUG (ScheduledTrigger-3435-thread-1) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1227507 DEBUG (ScheduledTrigger-3435-thread-2) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"617085e047c3fTdd76bgupy4z9rx7h8ho6o06re",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1714174564793407,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714174564793407],
   [junit4]   2>     "_dequeue_time_":1714175630857983,
   [junit4]   2>     "_enqueue_time_":1714175567744088,
   [junit4]   2>     "nodeNames":["127.0.0.1:49814_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1227508 DEBUG (ScheduledTrigger-3435-thread-2) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_triggerEQ]
   [junit4]   2> 1227508 INFO  (ScheduledTrigger-3435-thread-2) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 1227508 DEBUG (AutoscalingActionExecutor-3436-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"617085e047c3fTdd76bgupy4z9rx7h8ho6o06re",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1714174564793407,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714174564793407],
   [junit4]   2>     "_dequeue_time_":1714175630857983,
   [junit4]   2>     "_enqueue_time_":1714175567744088,
   [junit4]   2>     "nodeNames":["127.0.0.1:49814_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1227509 INFO  (AutoscalingActionExecutor-3436-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.TriggerIntegrationTest -- event: {
   [junit4]   2>   "id":"617085e047c3fTdd76bgupy4z9rx7h8ho6o06re",
   [junit4]   2>   "source":"node_added_triggerEQ",
   [junit4]   2>   "eventTime":1714174564793407,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1714174564793407],
   [junit4]   2>     "_dequeue_time_":1714175630857983,
   [junit4]   2>     "_enqueue_time_":1714175567744088,
   [junit4]   2>     "nodeNames":["127.0.0.1:49814_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1227510 DEBUG (AutoscalingActionExecutor-3436-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 5000ms
   [junit4]   2> 1227510 DEBUG (AutoscalingActionExecutor-3436-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerEQ after 5000ms
   [junit4]   2> 1227510 DEBUG (AutoscalingActionExecutor-3436-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 1 ms for event id=617085e047c3fTdd76bgupy4z9rx7h8ho6o06re
   [junit4]   2> 1227511 DEBUG (ScheduledTrigger-3435-thread-2) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEQ
   [junit4]   2> 1227511 DEBUG (ScheduledTrigger-3435-thread-2) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1232483 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[71CDD7A29492139B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> 1232508 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 1232509 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 8
   [junit4]   2> 1232509 DEBUG (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 1232510 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1232513 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1232513 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1232513 DEBUG (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1232515 INFO  (qtp215013857-10217) [n:127.0.0.1:39076_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1232521 INFO  (qtp215013857-10217) [n:127.0.0.1:39076_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 1232522 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:27:37+09:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1232524 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1232524 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1232524 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1232524 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2b121748{/solr,null,AVAILABLE}
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3438d102{HTTP/1.1,[http/1.1]}{127.0.0.1:50857}
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.Server Started @1236133ms
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50857}
   [junit4]   2> 1232525 ERROR (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1232525 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-09T17:41:50.354Z
   [junit4]   2> 1232527 INFO  (zkConnectionManagerCallback-2505-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1232528 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1232541 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47432/solr
   [junit4]   2> 1232543 INFO  (zkConnectionManagerCallback-2509-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1232547 INFO  (zkConnectionManagerCallback-2511-thread-1-processing-n:127.0.0.1:50857_solr) [n:127.0.0.1:50857_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1232572 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1232577 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1232582 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50857_solr
   [junit4]   2> 1232583 INFO  (zkCallback-2500-thread-1-processing-n:127.0.0.1:49814_solr) [n:127.0.0.1:49814_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1232585 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1232585 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1232585 INFO  (zkCallback-2482-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1232587 INFO  (zkCallback-2510-thread-1-processing-n:127.0.0.1:50857_solr) [n:127.0.0.1:50857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1232663 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50857.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1232681 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50857.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1232681 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50857.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ae6cc81
   [junit4]   2> 1232683 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [n:127.0.0.1:50857_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001/tempDir-001/node5/.
   [junit4]   2> 1232737 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== KILL OVERSEER 1
   [junit4]   2> 1232738 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3c84ff41{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1232739 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=842094954
   [junit4]   2> 1232739 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1232739 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@41d9ee81: rootName = solr_39076, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@610be30f
   [junit4]   2> 1232747 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1232747 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@387d7509: rootName = solr_39076, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@26224cdc
   [junit4]   2> 1232752 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1232752 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1b6824a0: rootName = solr_39076, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1378bd3a
   [junit4]   2> 1232755 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:39076_solr
   [junit4]   2> 1232757 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1232758 INFO  (zkCallback-2482-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1232758 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1232758 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1232758 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.Overseer Overseer (id=72169933809778694-127.0.0.1:39076_solr-n_0000000001) closing
   [junit4]   2> 1232759 INFO  (OverseerStateUpdate-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:39076_solr
   [junit4]   2> 1232759 WARN  (OverseerAutoScalingTriggerThread-72169933809778694-127.0.0.1:39076_solr-n_0000000001) [n:127.0.0.1:39076_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1232767 INFO  (zkCallback-2472-thread-1-processing-n:127.0.0.1:39076_solr) [n:127.0.0.1:39076_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1232767 DEBUG (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1232770 INFO  (zkCallback-2510-thread-1-processing-n:127.0.0.1:50857_solr) [n:127.0.0.1:50857_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1232770 INFO  (zkCallback-2500-thread-1-processing-n:127.0.0.1:49814_solr) [n:127.0.0.1:49814_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1232770 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42044_solr
   [junit4]   2> 1232771 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1bd9b33{/solr,null,UNAVAILABLE}
   [junit4]   2> 1232771 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1232772 INFO  (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.Overseer Overseer (id=72169933809778699-127.0.0.1:42044_solr-n_0000000002) starting
   [junit4]   2> 1232783 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1232785 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 1232786 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:50857_solr]
   [junit4]   2> 1232786 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
   [junit4]   2> 1232786 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1232790 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1232791 DEBUG (ScheduledTrigger-3446-thread-1) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1232792 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1233793 DEBUG (ScheduledTrigger-3446-thread-1) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1234795 DEBUG (ScheduledTrigger-3446-thread-2) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1235796 DEBUG (ScheduledTrigger-3446-thread-1) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1236797 DEBUG (ScheduledTrigger-3446-thread-1) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1237774 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== ADD TRIGGERS
   [junit4]   2> 1237779 INFO  (qtp1612969104-10272) [n:127.0.0.1:42044_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1237782 INFO  (qtp927268948-10301) [n:127.0.0.1:49814_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1237789 INFO  (qtp351874305-10340) [n:127.0.0.1:50857_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1237789 DEBUG (qtp351874305-10335) [n:127.0.0.1:50857_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1237790 INFO  (qtp351874305-10335) [n:127.0.0.1:50857_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=13
   [junit4]   2> 1237791 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 1237792 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:50857_solr]
   [junit4]   2> 1237793 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 1237793 INFO  (qtp1612969104-10274) [n:127.0.0.1:42044_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1237793 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:50857_solr]
   [junit4]   2> 1237793 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 1237795 INFO  (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
   [junit4]   2> 1237795 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 1237796 DEBUG (ScheduledTrigger-3446-thread-4) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
   [junit4]   2> 1237796 DEBUG (ScheduledTrigger-3446-thread-4) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1237797 DEBUG (ScheduledTrigger-3446-thread-3) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1237826 INFO  (qtp927268948-10303) [n:127.0.0.1:49814_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1237827 INFO  (qtp351874305-10340) [n:127.0.0.1:50857_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1237827 DEBUG (qtp351874305-10338) [n:127.0.0.1:50857_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1237829 INFO  (qtp351874305-10338) [n:127.0.0.1:50857_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=38
   [junit4]   2> 1237830 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 1237830 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:50857_solr]
   [junit4]   2> 1237831 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:50857_solr]
   [junit4]   2> 1237831 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
   [junit4]   2> 1237831 DEBUG (zkCallback-2490-thread-1-processing-n:127.0.0.1:42044_solr) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:42044_solr, 127.0.0.1:49814_solr, 127.0.0.1:50857_solr]
   [junit4]   2> 1237831 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 1237831 INFO  (qtp1612969104-10274) [n:127.0.0.1:42044_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1237833 INFO  (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
   [junit4]   2> 1237836 DEBUG (OverseerAutoScalingTriggerThread-72169933809778699-127.0.0.1:42044_solr-n_0000000002) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 1237836 DEBUG (ScheduledTrigger-3446-thread-2) [n:127.0.0.1:42044_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 3
   [junit4]   2> 1237839 INFO  (qtp1612969104-10274) [n:127.0.0.1:42044_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 1237840 INFO  (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[71CDD7A29492139B]) [    ] o.a.s.c.a.TriggerIntegrationTest ====== ADD NODE 1
   [junit4]  

[...truncated too long message...]

:core_node10 x:collection1_shard1_replica_n9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=767986259
   [junit4]   2> 1415285 INFO  (coreCloseExecutor-3622-thread-2) [n:127.0.0.1:35483_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testMetricTrigger.shard2.leader, tag=1693706137
   [junit4]   2> 1415304 INFO  (coreCloseExecutor-3627-thread-2) [n:127.0.0.1:52054_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testMetricTrigger.shard1.leader, tag=1992556257
   [junit4]   2> 1415306 INFO  (coreCloseExecutor-3621-thread-2) [n:127.0.0.1:57054_solr c:collection1 s:shard1 r:core_node16 x:collection1_shard1_replica_n15] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=658738278
   [junit4]   2> 1415315 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100662c24320023, likely client has closed socket
   [junit4]   2> 1415315 INFO  (jetty-closer-2455-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72169933809778711-127.0.0.1:52054_solr-n_0000000006) closing
   [junit4]   2> 1415318 INFO  (OverseerStateUpdate-72169933809778711-127.0.0.1:52054_solr-n_0000000006) [n:127.0.0.1:52054_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:52054_solr
   [junit4]   2> 1415328 INFO  (jetty-closer-2455-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6dadb644{/solr,null,UNAVAILABLE}
   [junit4]   2> 1415328 INFO  (jetty-closer-2455-thread-4) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1415332 WARN  (OverseerAutoScalingTriggerThread-72169933809778711-127.0.0.1:52054_solr-n_0000000006) [n:127.0.0.1:52054_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1415334 DEBUG (jetty-closer-2455-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1415342 INFO  (jetty-closer-2455-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44e1eb38{/solr,null,UNAVAILABLE}
   [junit4]   2> 1415343 INFO  (jetty-closer-2455-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5d781908{/solr,null,UNAVAILABLE}
   [junit4]   2> 1415343 INFO  (jetty-closer-2455-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1415343 INFO  (jetty-closer-2455-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1416170 WARN  (zkCallback-2658-thread-1-processing-n:127.0.0.1:46716_solr) [n:127.0.0.1:46716_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1416171 INFO  (jetty-closer-2455-thread-11) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4b11a4c0{/solr,null,UNAVAILABLE}
   [junit4]   2> 1416171 INFO  (jetty-closer-2455-thread-11) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1416769 INFO  (jetty-closer-2455-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2abfecb{/solr,null,UNAVAILABLE}
   [junit4]   2> 1416769 INFO  (jetty-closer-2455-thread-6) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1416772 ERROR (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1416772 INFO  (SUITE-TriggerIntegrationTest-seed#[71CDD7A29492139B]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47432 47432
   [junit4]   2> 1427653 INFO  (Thread-2911) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47432 47432
   [junit4]   2> 1427655 WARN  (Thread-2911) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 39 /solr/collections/collection1/terms/shard1
   [junit4]   2> 22 /solr/clusterprops.json
   [junit4]   2> 22 /solr/aliases.json
   [junit4]   2> 21 /solr/security.json
   [junit4]   2> 6 /solr/configs/conf
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard1
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard2
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 109 /solr/collections/collection1/state.json
   [junit4]   2> 22 /solr/clusterstate.json
   [junit4]   2> 13 /solr/collections/testMetricTrigger/state.json
   [junit4]   2> 6 /solr/autoscaling.json
   [junit4]   2> 3 /solr/overseer_elect/election/72169933809778742-127.0.0.1:63437_solr-n_0000000016
   [junit4]   2> 2 /solr/overseer_elect/election/72169933809778723-127.0.0.1:35483_solr-n_0000000010
   [junit4]   2> 2 /solr/overseer_elect/election/72169933809778729-127.0.0.1:42813_solr-n_0000000012
   [junit4]   2> 2 /solr/overseer_elect/election/72169933809778705-127.0.0.1:50857_solr-n_0000000004
   [junit4]   2> 2 /solr/overseer_elect/election/72169933809778714-127.0.0.1:54977_solr-n_0000000007
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72169933809778723-core_node10-n_0000000004
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72169933809778726-core_node4-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72169933809778736-127.0.0.1:57273_solr-n_0000000014
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72169933809778711-core_node8-n_0000000003
   [junit4]   2> 2 /solr/overseer_elect/election/72169933809778711-127.0.0.1:52054_solr-n_0000000006
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 22 /solr/live_nodes
   [junit4]   2> 22 /solr/collections
   [junit4]   2> 6 /solr/overseer/queue-work
   [junit4]   2> 6 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 6 /solr/overseer/queue
   [junit4]   2> 6 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/autoscaling/events/node_lost_triggerMR
   [junit4]   2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_71CDD7A29492139B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=Direct), x_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=929, maxMBSortInHeap=5.931696344499076, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@28f5095f), locale=ar-SA, timezone=Australia/NSW
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=93871112,total=272285696
   [junit4]   2> NOTE: All tests run in this JVM: [MergeStrategyTest, TestNamedUpdateProcessors, TestBlendedInfixSuggestions, ManagedSchemaRoundRobinCloudTest, OverseerTaskQueueTest, ExternalFileFieldSortTest, BadComponentTest, TestSortableTextField, TestBlobHandler, TestPushWriter, AtomicUpdateProcessorFactoryTest, CollectionStateFormat2Test, DocValuesNotIndexedTest, TestLegacyTerms, CurrencyFieldTypeTest, ExitableDirectoryReaderTest, CheckHdfsIndexTest, DistributedFacetPivotWhiteBoxTest, TestConfigOverlay, TestCollationFieldDocValues, UninvertDocValuesMergePolicyTest, TestCollapseQParserPlugin, LegacyCloudClusterPropTest, ZkStateWriterTest, DistanceFunctionTest, ConcurrentDeleteAndCreateCollectionTest, TestNumericRangeQuery32, CustomHighlightComponentTest, DOMUtilTest, TlogReplayBufferedWhileIndexingTest, TestBulkSchemaAPI, TestConfigSetsAPIZkFailure, DistribDocExpirationUpdateProcessorTest, TestSolrConfigHandlerConcurrent, TestMaxScoreQueryParser, TestFieldCacheSort, TestScoreJoinQPScore, SparseHLLTest, MissingSegmentRecoveryTest, PropertiesRequestHandlerTest, TestPKIAuthenticationPlugin, DeleteLastCustomShardedReplicaTest, PrimitiveFieldTypeTest, EchoParamsTest, TestDeleteCollectionOnDownNodes, AutoscalingHistoryHandlerTest, TestConfigsApi, FieldAnalysisRequestHandlerTest, MetricTriggerTest, InfoHandlerTest, RequestHandlersTest, RuleEngineTest, TestCodecSupport, ZkSolrClientTest, TestSolr4Spatial, TestWordDelimiterFilterFactory, TestJettySolrRunner, NodeLostTriggerTest, TestInitQParser, TestSolrCloudSnapshots, CloudMLTQParserTest, SpellCheckCollatorTest, AddBlockUpdateTest, DistributedFacetSimpleRefinementLongTailTest, ClassificationUpdateProcessorFactoryTest, TestExportWriter, SolrGangliaReporterTest, TestStressUserVersions, HdfsBasicDistributedZk2Test, SchemaApiFailureTest, TestCloudJSONFacetJoinDomain, TestCrossCoreJoin, TestDistributedSearch, TestGraphMLResponseWriter, CdcrVersionReplicationTest, ConnectionManagerTest, CoreSorterTest, TestFieldCollectionResource, DistributedQueryComponentOptimizationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, SuggesterWFSTTest, TestExpandComponent, AliasIntegrationTest, SolrShardReporterTest, TestValueSourceCache, DocExpirationUpdateProcessorFactoryTest, SolrMetricReporterTest, TestAnalyzedSuggestions, DocValuesMultiTest, TestRequestForwarding, OverseerTest, TestCoreDiscovery, XsltUpdateRequestHandlerTest, TestSegmentSorting, BooleanFieldTest, SoftAutoCommitTest, TestSerializedLuceneMatchVersion, PeerSyncTest, SolrCLIZkUtilsTest, TriggerIntegrationTest]
   [junit4] Completed [152/775 (1!)] on J1 in 207.04s, 14 tests, 1 failure <<< FAILURES!

[...truncated 50200 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1669 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1669/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory

Error Message:
expected:<5> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<0>
        at __randomizedtesting.SeedInfo.seed([A0AB985EA14A75C1:CD573CA31B028AC6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:274)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14173 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
   [junit4]   2> 4959391 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/init-core-data-001
   [junit4]   2> 4959392 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4959393 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 4959393 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001
   [junit4]   2> 4959393 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4959394 INFO  (Thread-10620) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4959394 INFO  (Thread-10620) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4959395 ERROR (Thread-10620) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4959493 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:63232
   [junit4]   2> 4959500 INFO  (zkConnectionManagerCallback-6679-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959508 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4959508 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4959508 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4959509 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4959509 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4959509 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 4959511 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4959511 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4959511 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 4959516 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@30158f9b{/solr,null,AVAILABLE}
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1bd591f0{/solr,null,AVAILABLE}
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7036491e{/solr,null,AVAILABLE}
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@45984720{HTTP/1.1,[http/1.1]}{127.0.0.1:57601}
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-1) [    ] o.e.j.s.Server Started @4968225ms
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57601}
   [junit4]   2> 4959517 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4218a472{HTTP/1.1,[http/1.1]}{127.0.0.1:45198}
   [junit4]   2> 4959517 ERROR (jetty-launcher-6676-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4959518 INFO  (jetty-launcher-6676-thread-2) [    ] o.e.j.s.Server Started @4968226ms
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45198}
   [junit4]   2> 4959520 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5f455e10{HTTP/1.1,[http/1.1]}{127.0.0.1:52848}
   [junit4]   2> 4959521 ERROR (jetty-launcher-6676-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4959520 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-3) [    ] o.e.j.s.Server Started @4968229ms
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52848}
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4959521 ERROR (jetty-launcher-6676-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4959522 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4959522 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T05:15:42.396Z
   [junit4]   2> 4959521 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T05:15:42.395Z
   [junit4]   2> 4959522 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T05:15:42.396Z
   [junit4]   2> 4959524 INFO  (zkConnectionManagerCallback-6681-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959524 INFO  (zkConnectionManagerCallback-6683-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959524 INFO  (zkConnectionManagerCallback-6685-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959524 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4959525 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4959525 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4959530 INFO  (jetty-launcher-6676-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63232/solr
   [junit4]   2> 4959530 INFO  (jetty-launcher-6676-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63232/solr
   [junit4]   2> 4959532 INFO  (zkConnectionManagerCallback-6693-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959532 INFO  (jetty-launcher-6676-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63232/solr
   [junit4]   2> 4959532 INFO  (zkConnectionManagerCallback-6691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959534 INFO  (zkConnectionManagerCallback-6697-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959534 INFO  (zkConnectionManagerCallback-6699-thread-1-processing-n:127.0.0.1:57601_solr) [n:127.0.0.1:57601_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959535 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10068a8661a0006, likely client has closed socket
   [junit4]   2> 4959536 INFO  (zkConnectionManagerCallback-6701-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959537 INFO  (zkConnectionManagerCallback-6703-thread-1-processing-n:127.0.0.1:45198_solr) [n:127.0.0.1:45198_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4959693 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4959694 INFO  (jetty-launcher-6676-thread-2) [n:127.0.0.1:45198_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4959694 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52848_solr
   [junit4]   2> 4959695 INFO  (jetty-launcher-6676-thread-2) [n:127.0.0.1:45198_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45198_solr
   [junit4]   2> 4959695 INFO  (jetty-launcher-6676-thread-1) [n:127.0.0.1:57601_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4959696 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer Overseer (id=72172666514702344-127.0.0.1:52848_solr-n_0000000000) starting
   [junit4]   2> 4959698 INFO  (zkCallback-6702-thread-1-processing-n:127.0.0.1:45198_solr) [n:127.0.0.1:45198_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4959701 INFO  (zkCallback-6700-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4959703 INFO  (zkCallback-6698-thread-1-processing-n:127.0.0.1:57601_solr) [n:127.0.0.1:57601_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4959705 INFO  (jetty-launcher-6676-thread-1) [n:127.0.0.1:57601_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57601_solr
   [junit4]   2> 4959706 INFO  (zkCallback-6698-thread-1-processing-n:127.0.0.1:57601_solr) [n:127.0.0.1:57601_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4959707 INFO  (zkCallback-6702-thread-1-processing-n:127.0.0.1:45198_solr) [n:127.0.0.1:45198_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4959707 INFO  (zkCallback-6700-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4959713 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52848_solr
   [junit4]   2> 4959716 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 4959716 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 4959717 INFO  (zkCallback-6700-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4959728 INFO  (zkCallback-6702-thread-1-processing-n:127.0.0.1:45198_solr) [n:127.0.0.1:45198_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4959743 INFO  (zkCallback-6698-thread-1-processing-n:127.0.0.1:57601_solr) [n:127.0.0.1:57601_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4959744 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 4959744 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:52848_solr, 127.0.0.1:57601_solr, 127.0.0.1:45198_solr]
   [junit4]   2> 4959744 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 4959744 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 4959746 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4959746 DEBUG (OverseerAutoScalingTriggerThread-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 4959748 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:57601_solr"} current state version: 0
   [junit4]   2> 4959748 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:57601_solr
   [junit4]   2> 4959748 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:52848_solr"} current state version: 0
   [junit4]   2> 4959748 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:52848_solr
   [junit4]   2> 4959757 DEBUG (ScheduledTrigger-10729-thread-1) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 4959878 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52848.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959881 INFO  (jetty-launcher-6676-thread-2) [n:127.0.0.1:45198_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45198.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959891 INFO  (jetty-launcher-6676-thread-1) [n:127.0.0.1:57601_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57601.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959929 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52848.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959929 INFO  (jetty-launcher-6676-thread-2) [n:127.0.0.1:45198_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45198.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959929 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52848.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959929 INFO  (jetty-launcher-6676-thread-2) [n:127.0.0.1:45198_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45198.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959930 INFO  (jetty-launcher-6676-thread-1) [n:127.0.0.1:57601_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57601.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959930 INFO  (jetty-launcher-6676-thread-1) [n:127.0.0.1:57601_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57601.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4959930 INFO  (jetty-launcher-6676-thread-3) [n:127.0.0.1:52848_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/.
   [junit4]   2> 4959930 INFO  (jetty-launcher-6676-thread-2) [n:127.0.0.1:45198_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/.
   [junit4]   2> 4959931 INFO  (jetty-launcher-6676-thread-1) [n:127.0.0.1:57601_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/.
   [junit4]   2> 4959999 INFO  (zkConnectionManagerCallback-6711-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4960003 INFO  (zkConnectionManagerCallback-6715-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4960005 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4960006 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63232/solr ready
   [junit4]   2> 4960012 INFO  (qtp1158603464-35053) [n:127.0.0.1:52848_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&name=AutoscalingHistoryHandlerTest_collection&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4960015 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection AutoscalingHistoryHandlerTest_collection
   [junit4]   2> 4960017 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/protwords.txt
   [junit4]   2> 4960020 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/managed-schema
   [junit4]   2> 4960023 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/solrconfig.xml
   [junit4]   2> 4960025 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/synonyms.txt
   [junit4]   2> 4960028 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/stopwords.txt
   [junit4]   2> 4960030 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/params.json
   [junit4]   2> 4960034 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_it.txt
   [junit4]   2> 4960037 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stemdict_nl.txt
   [junit4]   2> 4960040 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_no.txt
   [junit4]   2> 4960043 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_hy.txt
   [junit4]   2> 4960046 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_ca.txt
   [junit4]   2> 4960050 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_id.txt
   [junit4]   2> 4960054 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ga.txt
   [junit4]   2> 4960056 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_da.txt
   [junit4]   2> 4960059 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_hi.txt
   [junit4]   2> 4960061 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ja.txt
   [junit4]   2> 4960064 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_pt.txt
   [junit4]   2> 4960067 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_hu.txt
   [junit4]   2> 4960069 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_el.txt
   [junit4]   2> 4960072 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ru.txt
   [junit4]   2> 4960075 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_tr.txt
   [junit4]   2> 4960078 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ar.txt
   [junit4]   2> 4960080 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/userdict_ja.txt
   [junit4]   2> 4960083 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_eu.txt
   [junit4]   2> 4960086 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_it.txt
   [junit4]   2> 4960089 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_cz.txt
   [junit4]   2> 4960092 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stoptags_ja.txt
   [junit4]   2> 4960095 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_fr.txt
   [junit4]   2> 4960098 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_de.txt
   [junit4]   2> 4960100 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_fa.txt
   [junit4]   2> 4960103 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_ga.txt
   [junit4]   2> 4960105 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ca.txt
   [junit4]   2> 4960108 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_nl.txt
   [junit4]   2> 4960111 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_bg.txt
   [junit4]   2> 4960112 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_lv.txt
   [junit4]   2> 4960119 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_en.txt
   [junit4]   2> 4960122 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_es.txt
   [junit4]   2> 4960125 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ro.txt
   [junit4]   2> 4960128 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_th.txt
   [junit4]   2> 4960131 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_fr.txt
   [junit4]   2> 4960134 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_sv.txt
   [junit4]   2> 4960137 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_fi.txt
   [junit4]   2> 4960140 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/hyphenations_ga.txt
   [junit4]   2> 4960143 INFO  (OverseerThreadFactory-10731-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_gl.txt
   [junit4]   2> 4960155 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 4960155 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: AutoscalingHistoryHandlerTest_collection
   [junit4]   2> 4960155 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/AutoscalingHistoryHandlerTest_collection/state.json
   [junit4]   2> 4960261 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 4960261 INFO  (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 4960263 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{}}
   [junit4]   2> 4960263 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 4960265 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 4960265 INFO  (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 4960267 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 4960267 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>       "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 4960268 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 4960268 INFO  (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 4960270 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>       "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 4960270 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.SliceMutator New Slice: shard1:{
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "replicas":{
   [junit4]   2>     "core_node3":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>       "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node5":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>       "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"},
   [junit4]   2>     "core_node6":{
   [junit4]   2>       "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>       "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>       "state":"down",
   [junit4]   2>       "node_name":null,
   [junit4]   2>       "type":"NRT"}}}
   [junit4]   2> 4960370 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/AutoscalingHistoryHandlerTest_collection/state.json version: 0
   [junit4]   2> 4960472 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=AutoscalingHistoryHandlerTest_collection.AUTOCREATED&newCollection=true&name=AutoscalingHistoryHandlerTest_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=AutoscalingHistoryHandlerTest_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4960473 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=AutoscalingHistoryHandlerTest_collection.AUTOCREATED&newCollection=true&name=AutoscalingHistoryHandlerTest_collection_shard1_replica_n2&action=CREATE&numShards=1&collection=AutoscalingHistoryHandlerTest_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4960473 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4960473 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=AutoscalingHistoryHandlerTest_collection.AUTOCREATED&newCollection=true&name=AutoscalingHistoryHandlerTest_collection_shard1_replica_n4&action=CREATE&numShards=1&collection=AutoscalingHistoryHandlerTest_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4960473 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4960473 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>   "core_node_name":"core_node6",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>   "node_name":"127.0.0.1:45198_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>   "core_node_name":"core_node6",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>   "node_name":"127.0.0.1:45198_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node6:{"core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4","base_url":"http://127.0.0.1:45198/solr","node_name":"127.0.0.1:45198_solr","state":"down","type":"NRT"}
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(AutoscalingHistoryHandlerTest_collection//collections/AutoscalingHistoryHandlerTest_collection/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>           "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>           "node_name":"127.0.0.1:45198_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"}}}}}
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>   "core_node_name":"core_node5",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>   "node_name":"127.0.0.1:57601_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>   "core_node_name":"core_node5",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>   "node_name":"127.0.0.1:57601_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node5:{"core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2","base_url":"http://127.0.0.1:57601/solr","node_name":"127.0.0.1:57601_solr","state":"down","type":"NRT"}
   [junit4]   2> 4960478 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(AutoscalingHistoryHandlerTest_collection//collections/AutoscalingHistoryHandlerTest_collection/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "node_name":null,
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>           "node_name":"127.0.0.1:57601_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>           "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>           "node_name":"127.0.0.1:45198_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"}}}}}
   [junit4]   2> 4960479 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>   "node_name":"127.0.0.1:52848_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 4960479 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>   "node_name":"127.0.0.1:52848_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"AutoscalingHistoryHandlerTest_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 4960479 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node3:{"core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1","base_url":"http://127.0.0.1:52848/solr","node_name":"127.0.0.1:52848_solr","state":"down","type":"NRT"}
   [junit4]   2> 4960479 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(AutoscalingHistoryHandlerTest_collection//collections/AutoscalingHistoryHandlerTest_collection/state.json/1)={
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "replicationFactor":"3",
   [junit4]   2>   "router":{"name":"compositeId"},
   [junit4]   2>   "maxShardsPerNode":"1",
   [junit4]   2>   "autoAddReplicas":"false",
   [junit4]   2>   "nrtReplicas":"3",
   [junit4]   2>   "tlogReplicas":"0",
   [junit4]   2>   "shards":{"shard1":{
   [junit4]   2>       "range":"80000000-7fffffff",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "replicas":{
   [junit4]   2>         "core_node3":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
   [junit4]   2>           "base_url":"http://127.0.0.1:52848/solr",
   [junit4]   2>           "node_name":"127.0.0.1:52848_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node5":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
   [junit4]   2>           "base_url":"http://127.0.0.1:57601/solr",
   [junit4]   2>           "node_name":"127.0.0.1:57601_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"},
   [junit4]   2>         "core_node6":{
   [junit4]   2>           "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
   [junit4]   2>           "base_url":"http://127.0.0.1:45198/solr",
   [junit4]   2>           "node_name":"127.0.0.1:45198_solr",
   [junit4]   2>           "state":"down",
   [junit4]   2>           "type":"NRT"}}}}}
   [junit4]   2> 4960580 DEBUG (OverseerStateUpdate-72172666514702344-127.0.0.1:52848_solr-n_0000000000) [n:127.0.0.1:52848_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/AutoscalingHistoryHandlerTest_collection/state.json version: 1
   [junit4]   2> 4960581 INFO  (zkCallback-6702-thread-1-processing-n:127.0.0.1:45198_solr) [n:127.0.0.1:45198_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 4960581 INFO  (zkCallback-6698-thread-1-processing-n:127.0.0.1:57601_solr) [n:127.0.0.1:57601_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 4960581 INFO  (zkCallback-6702-thread-2-processing-n:127.0.0.1:45198_solr) [n:127.0.0.1:45198_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 4960581 INFO  (zkCallback-6700-thread-1-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 4960581 INFO  (zkCallback-6698-thread-2-processing-n:127.0.0.1:57601_solr) [n:127.0.0.1:57601_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 4960581 INFO  (zkCallback-6700-thread-2-processing-n:127.0.0.1:52848_solr) [n:127.0.0.1:52848_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 4960763 DEBUG (ScheduledTrigger-10729-thread-1) [n:127.0.0.1:52848_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 4961487 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 4961487 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 4961487 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 4961488 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 4961487 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 4961488 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 4961488 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/extraction/lib
   [junit4]   2> 4961488 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/extraction/lib
   [junit4]   2> 4961488 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist
   [junit4]   2> 4961489 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist
   [junit4]   2> 4961489 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/clustering/lib
   [junit4]   2> 4961489 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/clustering/lib
   [junit4]   2> 4961489 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist
   [junit4]   2> 4961489 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist
   [junit4]   2> 4961489 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/langid/lib
   [junit4]   2> 4961489 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/langid/lib
   [junit4]   2> 4961489 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist
   [junit4]   2> 4961489 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist
   [junit4]   2> 4961490 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../contrib/velocity/lib
   [junit4]   2> 4961490 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../contrib/velocity/lib
   [junit4]   2> 4961490 WARN  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node1/AutoscalingHistoryHandlerTest_collection_shard1_replica_n2/../../../../dist
   [junit4]   2> 4961490 WARN  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node2/AutoscalingHistoryHandlerTest_collection_shard1_replica_n4/../../../../dist
   [junit4]   2> 4961489 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 4961490 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist
   [junit4]   2> 4961490 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 4961490 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist
   [junit4]   2> 4961491 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 4961491 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist
   [junit4]   2> 4961491 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 4961491 WARN  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/../../../../dist
   [junit4]   2> 4961494 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4961494 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4961495 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4961527 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.s.IndexSchema [AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] Schema name=default-config
   [junit4]   2> 4961527 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.s.IndexSchema [AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] Schema name=default-config
   [junit4]   2> 4961528 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.s.IndexSchema [AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 4961704 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 4961712 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 4961714 INFO  (qtp783088425-35057) [n:127.0.0.1:45198_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node6 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n4] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 4961758 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'AutoscalingHistoryHandlerTest_collection_shard1_replica_n1' using configuration from collection AutoscalingHistoryHandlerTest_collection, trusted=true
   [junit4]   2> 4961759 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52848.solr.core.AutoscalingHistoryHandlerTest_collection.shard1.replica_n1' (registry 'solr.core.AutoscalingHistoryHandlerTest_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68990191
   [junit4]   2> 4961759 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4961759 INFO  (qtp1158603464-35058) [n:127.0.0.1:52848_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrCore [[AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/AutoscalingHistoryHandlerTest_collection_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001/tempDir-001/node3/./AutoscalingHistoryHandlerTest_collection_shard1_replica_n1/data/]
   [junit4]   2> 4961762 INFO  (qtp1310034775-35045) [n:127.0.0.1:57601_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node5 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'AutoscalingHistoryHandlerTest_collection_shard1_replica_n2' using configuration from collection AutoscalingHistoryHandlerTest_collection, trusted=true
   [junit4]   2> 4961763 INFO  (qtp1310034775-35

[...truncated too long message...]

AVAILABLE}
   [junit4]   2> 4995989 INFO  (jetty-closer-6677-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 4995989 INFO  (zkCallback-6722-thread-1-processing-n:127.0.0.1:61532_solr) [n:127.0.0.1:61532_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61532_solr
   [junit4]   2> 4995995 INFO  (coreCloseExecutor-10799-thread-2) [n:127.0.0.1:61532_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node8 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n7] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.AutoscalingHistoryHandlerTest_collection.shard1.leader, tag=1492107441
   [junit4]   2> 4996003 INFO  (jetty-closer-6677-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5e27faa2{/solr,null,UNAVAILABLE}
   [junit4]   2> 4996006 INFO  (jetty-closer-6677-thread-3) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 4996007 ERROR (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4996008 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[A0AB985EA14A75C1]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:63232 63232
   [junit4]   2> 5005842 INFO  (Thread-10620) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:63232 63232
   [junit4]   2> 5005846 WARN  (Thread-10620) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 22 /solr/collections/.system/terms/shard1
   [junit4]   2> 17 /solr/collections/AutoscalingHistoryHandlerTest_collection/terms/shard1
   [junit4]   2> 13 /solr/aliases.json
   [junit4]   2> 13 /solr/clusterprops.json
   [junit4]   2> 5 /solr/configs/.system
   [junit4]   2> 5 /solr/configs/.system/managed-schema
   [junit4]   2> 4 /solr/security.json
   [junit4]   2> 4 /solr/configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/managed-schema
   [junit4]   2> 4 /solr/configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 59 /solr/collections/.system/state.json
   [junit4]   2> 40 /solr/collections/AutoscalingHistoryHandlerTest_collection/state.json
   [junit4]   2> 13 /solr/clusterstate.json
   [junit4]   2> 2 /solr/collections/AutoscalingHistoryHandlerTest_collection/leader_elect/shard1/election/72172666514702343-core_node5-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72172666514702345-127.0.0.1:45198_solr-n_0000000001
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2> 2 /solr/collections/.system/leader_elect/shard1/election/72172666514702343-core_node5-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 13 /solr/live_nodes
   [junit4]   2> 13 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/autoscaling/events/AutoscalingHistoryHandlerTest_node_lost_trigger
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_A0AB985EA14A75C1-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@40d14364), locale=bg, timezone=America/Havana
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=130440352,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestMaxTokenLenTokenizer, TestFieldTypeCollectionResource, BasicDistributedZk2Test, TestPayloadScoreQParserPlugin, TestManagedSchemaThreadSafety, TestLeaderInitiatedRecoveryThread, HighlighterConfigTest, TestSolrCloudSnapshots, SoftAutoCommitTest, MissingSegmentRecoveryTest, TestSolrConfigHandlerConcurrent, ZkSolrClientTest, TestWordDelimiterFilterFactory, CleanupOldIndexTest, TestImpersonationWithHadoopAuth, TestSolrConfigHandler, RollingRestartTest, TestExtendedDismaxParser, TestFieldSortValues, ReplicationFactorTest, URLClassifyProcessorTest, ConnectionManagerTest, TestComplexPhraseQParserPlugin, SystemLogListenerTest, SolrShardReporterTest, TestAnalyzedSuggestions, TestSort, SpellCheckCollatorTest, UpdateParamsTest, TestClassicSimilarityFactory, TestLegacyFieldReuse, CollectionsAPIAsyncDistributedZkTest, TestSolr4Spatial, OverseerCollectionConfigSetProcessorTest, PropertiesRequestHandlerTest, DocValuesMultiTest, LeaderFailureAfterFreshStartTest, DistributedSpellCheckComponentTest, BasicAuthIntegrationTest, SaslZkACLProviderTest, TestPullReplicaErrorHandling, TestSegmentSorting, TestNumericRangeQuery32, TestFastWriter, BasicFunctionalityTest, TestJettySolrRunner, ZkStateWriterTest, TestStressLucene, SimpleCollectionCreateDeleteTest, IndexSchemaTest, TestReplicationHandlerBackup, TestDynamicLoading, CustomHighlightComponentTest, V2ApiIntegrationTest, TestCloudPseudoReturnFields, TestSolrDeletionPolicy2, DirectUpdateHandlerTest, TestRequestStatusCollectionAPI, TestDistributedSearch, JavabinLoaderTest, TestSha256AuthenticationProvider, AliasIntegrationTest, DOMUtilTest, TestMultiWordSynonyms, SystemInfoHandlerTest, TestValueSourceCache, FullSolrCloudDistribCmdsTest, TestConfigSetsAPIZkFailure, MultiThreadedOCPTest, TestCoreDiscovery, SparseHLLTest, BlobRepositoryCloudTest, TestSubQueryTransformer, AutoAddReplicasPlanActionTest, TestUseDocValuesAsStored2, TestConfigSetProperties, TestCloudJSONFacetJoinDomain, GraphQueryTest, InfoHandlerTest, DisMaxRequestHandlerTest, HdfsCollectionsAPIDistributedZkTest, CheckHdfsIndexTest, TestLMDirichletSimilarityFactory, CloudMLTQParserTest, TestJsonFacets, TestFieldCacheSort, TestSimDistributedQueue, TestSolrIndexConfig, SimpleFacetsTest, RequestLoggingTest, DocValuesNotIndexedTest, SolrMetricReporterTest, TestRTimerTree, TestRawTransformer, TestReversedWildcardFilterFactory, BlockJoinFacetRandomTest, BufferStoreTest, TestSimpleTrackingShardHandler, TestFuzzyAnalyzedSuggestions, TestPushWriter, TestMacroExpander, HLLSerializationTest, HdfsChaosMonkeyNothingIsSafeTest, DistributedExpandComponentTest, TestCollectionsAPIViaSolrCloudCluster, TestRestoreCore, ChaosMonkeyNothingIsSafeTest, HdfsWriteToMultipleCollectionsTest, TestDynamicFieldCollectionResource, DistributedIntervalFacetingTest, RuleEngineTest, BadComponentTest, HttpPartitionTest, TestQuerySenderListener, TestStressReorder, SolrCLIZkUtilsTest, TestSolr4Spatial2, ConcurrentDeleteAndCreateCollectionTest, AddBlockUpdateTest, TestSolrCloudWithSecureImpersonation, TestSolrQueryParser, TestGraphTermsQParserPlugin, TestIndexingPerformance, CdcrReplicationHandlerTest, TestCloudDeleteByQuery, TestLRUCache, BlockCacheTest, TestComputePlanAction, TestRandomCollapseQParserPlugin, TestDistributedMissingSort, TestDownShardTolerantSearch, CurrencyFieldTypeTest, CdcrBootstrapTest, TestMaxScoreQueryParser, ExternalFileFieldSortTest, ZkControllerTest, TestBlendedInfixSuggestions, TestBlobHandler, NotRequiredUniqueKeyTest, SolrJmxReporterCloudTest, DefaultValueUpdateProcessorTest, TestConfigSetsAPI, TestZkAclsWithHadoopAuth, TestSortableTextField, TestNumericTokenStream, TestLegacyFieldCache, EchoParamsTest, ClusterStateUpdateTest, HdfsTlogReplayBufferedWhileIndexingTest, SuggesterWFSTTest, TestExecutePlanAction, PeerSyncWithIndexFingerprintCachingTest, ChaosMonkeySafeLeaderTest, TestMiniSolrCloudClusterSSL, TestBulkSchemaAPI, PeerSyncTest, TestNestedDocsSort, TestInPlaceUpdatesStandalone, DeleteLastCustomShardedReplicaTest, DistribDocExpirationUpdateProcessorTest, TestSimpleQParserPlugin, OverseerRolesTest, TestExactStatsCache, TestGraphMLResponseWriter, SolrXmlInZkTest, QueryResultKeyTest, ConcurrentCreateRoutedAliasTest, SchemaApiFailureTest, TestHdfsUpdateLog, TestRequestForwarding, TestSchemaNameResource, TestLuceneIndexBackCompat, TestSolrCoreProperties, SuggestComponentTest, TriggerIntegrationTest, LoggingHandlerTest, FileBasedSpellCheckerTest, BJQParserTest, DistanceUnitsTest, TestInfoStreamLogging, TestRTGBase, SolrGangliaReporterTest, TestCollapseQParserPlugin, SpatialHeatmapFacetsTest, VersionInfoTest, RequestHandlersTest, TestUpdate, OverseerTaskQueueTest, MinimalSchemaTest, TestCloudManagedSchema, PKIAuthenticationIntegrationTest, HealthCheckHandlerTest, HttpPartitionOnCommitTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, MoveReplicaHDFSTest, OverseerModifyCollectionTest, RecoveryAfterSoftCommitTest, RestartWhileUpdatingTest, SSLMigrationTest, SolrCloudExampleTest, TestCloudPivotFacet, TestClusterProperties, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestLeaderElectionWithEmptyReplica, TestRandomFlRTGCloud, TestRandomRequestDistribution, TestRebalanceLeaders, TestSizeLimitedDistributedMap, TestTlogReplica, TestUtilizeNode, VMParamsZkACLAndCredentialsProvidersTest, ZkShardTermsTest, AssignTest, CollectionReloadTest, CollectionsAPIDistributedZkTest, ComputePlanActionTest, HdfsAutoAddReplicasIntegrationTest, TestDistribStateManager, TestNodeAddedTrigger, TestNodeLostTrigger, TestPolicyCloud, BaseCdcrDistributedZkTest, CdcrBidirectionalTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest, HdfsBasicDistributedZk2Test, HdfsNNFailoverTest, HdfsRestartWhileUpdatingTest, HdfsSyncSliceTest, TestClusterStateMutator, ImplicitSnitchTest, RulesTest, TestCoreBackup, TestReqParamsAPI, TestSQLHandlerNonCloud, V2StandaloneTest, AutoscalingHistoryHandlerTest]
   [junit4] Completed [635/775 (1!)] on J1 in 46.50s, 1 test, 1 failure <<< FAILURES!

[...truncated 48589 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1670 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1670/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  org.apache.solr.cloud.TestPullReplica.testRealTimeGet

Error Message:
Unexpected replica count null Live Nodes: [127.0.0.1:48386_solr, 127.0.0.1:48870_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Unexpected replica count
null
Live Nodes: [127.0.0.1:48386_solr, 127.0.0.1:48870_solr]
Last available state: null
        at __randomizedtesting.SeedInfo.seed([BE99A5499E4E409F:E6F4504A3CF4EE56]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
        at org.apache.solr.cloud.TestPullReplica.testRealTimeGet(TestPullReplica.java:343)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud:     1) Thread[id=4626, name=qtp706262109-4626, state=TIMED_WAITING, group=TGRP-TestTolerantUpdateProcessorRandomCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud:
   1) Thread[id=4626, name=qtp706262109-4626, state=TIMED_WAITING, group=TGRP-TestTolerantUpdateProcessorRandomCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([BE99A5499E4E409F]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4626, name=qtp706262109-4626, state=TIMED_WAITING, group=TGRP-TestTolerantUpdateProcessorRandomCloud]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4626, name=qtp706262109-4626, state=TIMED_WAITING, group=TGRP-TestTolerantUpdateProcessorRandomCloud]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([BE99A5499E4E409F]:0)




Build Log:
[...truncated 12222 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/init-core-data-001
   [junit4]   2> 243853 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 243854 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-9182 - causes OOM)
   [junit4]   2> 243854 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 243854 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.TestTolerantUpdateProcessorRandomCloud Configuring cluster: servers=7, shards=3, repfactor=2
   [junit4]   2> 243855 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 7 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001
   [junit4]   2> 243855 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 243855 INFO  (Thread-2429) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 243855 INFO  (Thread-2429) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 243857 ERROR (Thread-2429) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 243955 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64869
   [junit4]   2> 243958 INFO  (zkConnectionManagerCallback-589-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 243962 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243962 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243963 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243963 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243964 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243964 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243965 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243965 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243965 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 243965 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28f292f4{/solr,null,AVAILABLE}
   [junit4]   2> 243967 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4e8e2249{HTTP/1.1,[http/1.1]}{127.0.0.1:44235}
   [junit4]   2> 243967 INFO  (jetty-launcher-586-thread-1) [    ] o.e.j.s.Server Started @246484ms
   [junit4]   2> 243967 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44235}
   [junit4]   2> 243968 ERROR (jetty-launcher-586-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243971 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243971 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243971 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243971 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.248Z
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 243971 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1dc8ab1e{/solr,null,AVAILABLE}
   [junit4]   2> 243972 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7b313e9c{/solr,null,AVAILABLE}
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.AbstractConnector Started ServerConnector@34d5dbee{HTTP/1.1,[http/1.1]}{127.0.0.1:40451}
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.e.j.s.Server Started @246489ms
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5398bb27{HTTP/1.1,[http/1.1]}{127.0.0.1:56212}
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40451}
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-4) [    ] o.e.j.s.Server Started @246489ms
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56212}
   [junit4]   2> 243973 ERROR (jetty-launcher-586-thread-7) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.250Z
   [junit4]   2> 243973 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 243974 ERROR (jetty-launcher-586-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243974 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243974 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243974 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243974 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.251Z
   [junit4]   2> 243969 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243974 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243974 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 243968 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243976 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243976 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 243976 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243976 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243977 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 243977 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@44928eb{/solr,null,AVAILABLE}
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33aa2fd8{HTTP/1.1,[http/1.1]}{127.0.0.1:37059}
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.e.j.s.Server Started @246494ms
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@716f08d2{/solr,null,AVAILABLE}
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37059}
   [junit4]   2> 243978 ERROR (jetty-launcher-586-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@107bc259{HTTP/1.1,[http/1.1]}{127.0.0.1:53714}
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.255Z
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.e.j.s.Server Started @246495ms
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53714}
   [junit4]   2> 243978 ERROR (jetty-launcher-586-thread-5) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243978 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.255Z
   [junit4]   2> 243994 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 243994 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 243994 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 243994 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@77558503{/solr,null,AVAILABLE}
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1cfd0d9a{HTTP/1.1,[http/1.1]}{127.0.0.1:33874}
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.e.j.s.Server Started @246511ms
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33874}
   [junit4]   2> 243995 ERROR (jetty-launcher-586-thread-6) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243995 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.272Z
   [junit4]   2> 243996 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14679af6{/solr,null,AVAILABLE}
   [junit4]   2> 243996 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@da99726{HTTP/1.1,[http/1.1]}{127.0.0.1:58461}
   [junit4]   2> 243996 INFO  (jetty-launcher-586-thread-2) [    ] o.e.j.s.Server Started @246513ms
   [junit4]   2> 243996 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58461}
   [junit4]   2> 243997 ERROR (jetty-launcher-586-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 243997 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 243997 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 243997 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 243997 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-10T15:59:09.274Z
   [junit4]   2> 243998 INFO  (zkConnectionManagerCallback-601-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 243999 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 243999 INFO  (zkConnectionManagerCallback-593-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244000 INFO  (zkConnectionManagerCallback-595-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244001 INFO  (zkConnectionManagerCallback-591-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244002 INFO  (zkConnectionManagerCallback-603-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244002 INFO  (zkConnectionManagerCallback-597-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244002 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 244003 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 244003 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 244003 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 244011 INFO  (zkConnectionManagerCallback-599-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244011 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 244011 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006af641290005, likely client has closed socket
   [junit4]   2> 244012 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006af641290001, likely client has closed socket
   [junit4]   2> 244012 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006af641290002, likely client has closed socket
   [junit4]   2> 244013 INFO  (jetty-launcher-586-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244013 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 244024 INFO  (jetty-launcher-586-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244024 INFO  (jetty-launcher-586-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244026 INFO  (jetty-launcher-586-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244028 INFO  (jetty-launcher-586-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244031 INFO  (zkConnectionManagerCallback-613-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244032 INFO  (zkConnectionManagerCallback-618-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244033 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006af641290003, likely client has closed socket
   [junit4]   2> 244032 INFO  (zkConnectionManagerCallback-608-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244034 INFO  (zkConnectionManagerCallback-623-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244035 INFO  (jetty-launcher-586-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244036 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006af64129000c, likely client has closed socket
   [junit4]   2> 244038 INFO  (zkConnectionManagerCallback-627-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244038 INFO  (zkConnectionManagerCallback-629-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244039 INFO  (zkConnectionManagerCallback-620-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244040 INFO  (zkConnectionManagerCallback-633-thread-1-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244040 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006af64129000d, likely client has closed socket
   [junit4]   2> 244040 INFO  (jetty-launcher-586-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64869/solr
   [junit4]   2> 244041 INFO  (zkConnectionManagerCallback-631-thread-1-processing-n:127.0.0.1:37059_solr) [n:127.0.0.1:37059_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244042 INFO  (zkConnectionManagerCallback-639-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244044 INFO  (zkConnectionManagerCallback-641-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244046 INFO  (zkConnectionManagerCallback-643-thread-1-processing-n:127.0.0.1:44235_solr) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244042 INFO  (zkConnectionManagerCallback-636-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244048 INFO  (zkConnectionManagerCallback-645-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 244296 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244298 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33874_solr
   [junit4]   2> 244299 INFO  (jetty-launcher-586-thread-5) [n:127.0.0.1:53714_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244301 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.c.Overseer Overseer (id=72175199925633038-127.0.0.1:33874_solr-n_0000000000) starting
   [junit4]   2> 244302 INFO  (jetty-launcher-586-thread-5) [n:127.0.0.1:53714_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53714_solr
   [junit4]   2> 244305 INFO  (jetty-launcher-586-thread-4) [n:127.0.0.1:56212_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244305 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 244306 INFO  (zkCallback-628-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 244306 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 244306 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 244307 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 244309 INFO  (jetty-launcher-586-thread-4) [n:127.0.0.1:56212_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56212_solr
   [junit4]   2> 244310 INFO  (jetty-launcher-586-thread-7) [n:127.0.0.1:40451_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244311 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 244312 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 244312 INFO  (zkCallback-628-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 244313 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 244315 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244320 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58461_solr
   [junit4]   2> 244330 INFO  (zkCallback-632-thread-1-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 244343 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 244358 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 244361 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 244365 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 244366 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33874_solr
   [junit4]   2> 244366 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 244380 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 244380 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 244383 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 244383 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 244383 INFO  (zkCallback-628-thread-2-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 244386 INFO  (jetty-launcher-586-thread-7) [n:127.0.0.1:40451_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40451_solr
   [junit4]   2> 244397 INFO  (jetty-launcher-586-thread-5) [n:127.0.0.1:53714_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53714.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244399 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 244399 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 244405 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58461.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244409 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58461.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244409 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58461.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244409 INFO  (jetty-launcher-586-thread-2) [n:127.0.0.1:58461_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node2/.
   [junit4]   2> 244409 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 244414 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 244414 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 244418 INFO  (zkCallback-628-thread-2-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 244423 INFO  (jetty-launcher-586-thread-5) [n:127.0.0.1:53714_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53714.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244428 INFO  (jetty-launcher-586-thread-5) [n:127.0.0.1:53714_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53714.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244429 INFO  (jetty-launcher-586-thread-5) [n:127.0.0.1:53714_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node5/.
   [junit4]   2> 244433 INFO  (zkCallback-630-thread-1-processing-n:127.0.0.1:37059_solr) [n:127.0.0.1:37059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 244434 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244436 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37059_solr
   [junit4]   2> 244476 INFO  (jetty-launcher-586-thread-4) [n:127.0.0.1:56212_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244482 INFO  (jetty-launcher-586-thread-4) [n:127.0.0.1:56212_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244482 INFO  (jetty-launcher-586-thread-4) [n:127.0.0.1:56212_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244483 INFO  (jetty-launcher-586-thread-4) [n:127.0.0.1:56212_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node3/.
   [junit4]   2> 244494 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 244494 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 244494 INFO  (zkCallback-628-thread-2-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 244503 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 244503 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 244503 INFO  (zkCallback-630-thread-1-processing-n:127.0.0.1:37059_solr) [n:127.0.0.1:37059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 244509 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 244512 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 244514 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44235_solr
   [junit4]   2> 244522 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244522 INFO  (zkCallback-630-thread-1-processing-n:127.0.0.1:37059_solr) [n:127.0.0.1:37059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244522 INFO  (zkCallback-642-thread-1-processing-n:127.0.0.1:44235_solr) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244522 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244523 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244523 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244526 INFO  (zkCallback-628-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 244539 INFO  (jetty-launcher-586-thread-7) [n:127.0.0.1:40451_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40451.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244589 INFO  (jetty-launcher-586-thread-7) [n:127.0.0.1:40451_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40451.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244590 INFO  (jetty-launcher-586-thread-7) [n:127.0.0.1:40451_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40451.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244590 INFO  (jetty-launcher-586-thread-7) [n:127.0.0.1:40451_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node6/.
   [junit4]   2> 244602 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37059.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244616 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44235.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244621 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44235.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244622 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44235.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244623 INFO  (jetty-launcher-586-thread-1) [n:127.0.0.1:44235_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node1/.
   [junit4]   2> 244624 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37059.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244624 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37059.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244625 INFO  (jetty-launcher-586-thread-3) [n:127.0.0.1:37059_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node4/.
   [junit4]   2> 244869 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33874.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244879 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33874.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244879 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33874.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 244880 INFO  (jetty-launcher-586-thread-6) [n:127.0.0.1:33874_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node7/.
   [junit4]   2> 245028 INFO  (zkConnectionManagerCallback-661-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 245033 INFO  (zkConnectionManagerCallback-665-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 245035 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 245036 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64869/solr ready
   [junit4]   2> 245618 INFO  (qtp198913640-4653) [n:127.0.0.1:40451_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&collection.configName=solrCloudCollectionConfig&name=test_col&nrtReplicas=2&action=CREATE&numShards=3&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 245629 INFO  (OverseerThreadFactory-1514-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 245754 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_col_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:40451/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 245758 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_col_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:56212/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 245762 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"test_col_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:33874/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 245766 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"test_col_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:44235/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 245769 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "core":"test_col_shard3_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:53714/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 245772 INFO  (OverseerStateUpdate-72175199925633038-127.0.0.1:33874_solr-n_0000000000) [n:127.0.0.1:33874_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test_col",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "core":"test_col_shard3_replica_n10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:58461/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 245979 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&coreNodeName=core_node3&name=test_col_shard1_replica_n1&action=CREATE&numShards=3&shard=shard1&wt=javabin
   [junit4]   2> 245980 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&coreNodeName=core_node9&name=test_col_shard2_replica_n6&action=CREATE&numShards=3&shard=shard2&wt=javabin
   [junit4]   2> 245980 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 245980 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 245982 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&coreNodeName=core_node7&name=test_col_shard2_replica_n4&action=CREATE&numShards=3&shard=shard2&wt=javabin
   [junit4]   2> 245982 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 245982 INFO  (qtp2002569912-4640) [n:127.0.0.1:56212_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&coreNodeName=core_node5&name=test_col_shard1_replica_n2&action=CREATE&numShards=3&shard=shard1&wt=javabin
   [junit4]   2> 245983 INFO  (qtp2002569912-4640) [n:127.0.0.1:56212_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 245984 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&coreNodeName=core_node11&name=test_col_shard3_replica_n8&action=CREATE&numShards=3&shard=shard3&wt=javabin
   [junit4]   2> 245984 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 245988 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=test_col&version=2&replicaType=NRT&property.config=solrconfig-distrib-update-processor-chains.xml&property.schema=schema15.xml&coreNodeName=core_node12&name=test_col_shard3_replica_n10&action=CREATE&numShards=3&shard=shard3&wt=javabin
   [junit4]   2> 245988 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 246105 INFO  (zkCallback-628-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-628-thread-2-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-640-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:40451_solr) [n:127.0.0.1:40451_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246105 INFO  (zkCallback-632-thread-1-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246106 INFO  (zkCallback-642-thread-1-processing-n:127.0.0.1:44235_solr) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246106 INFO  (zkCallback-642-thread-2-processing-n:127.0.0.1:44235_solr) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246106 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 246106 INFO  (zkCallback-644-thread-2-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247055 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 247056 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 247056 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 247072 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
   [junit4]   2> 247072 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
   [junit4]   2> 247075 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 247075 INFO  (qtp2002569912-4640) [n:127.0.0.1:56212_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 247076 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 247090 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
   [junit4]   2> 247091 INFO  (qtp2002569912-4640) [n:127.0.0.1:56212_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
   [junit4]   2> 247091 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
   [junit4]   2> 247095 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.IndexSchema [test_col_shard2_replica_n6] Schema name=test
   [junit4]   2> 247097 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.IndexSchema [test_col_shard2_replica_n4] Schema name=test
   [junit4]   2> 247101 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
   [junit4]   2> 247150 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema [test_col_shard1_replica_n1] Schema name=test
   [junit4]   2> 247196 INFO  (qtp2002569912-4640) [n:127.0.0.1:56212_solr c:test_col s:shard1 r:core_node5 x:test_col_shard1_replica_n2] o.a.s.s.IndexSchema [test_col_shard1_replica_n2] Schema name=test
   [junit4]   2> 247196 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.s.IndexSchema [test_col_shard3_replica_n8] Schema name=test
   [junit4]   2> 247209 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.s.IndexSchema [test_col_shard3_replica_n10] Schema name=test
   [junit4]   2> 247245 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 247335 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 247371 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 247382 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml to /configs/solrCloudCollectionConfig/schema15.xml.bak
   [junit4]   2> 247384 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
   [junit4]   2> 247385 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml no longer exists.
   [junit4]   2> 247385 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica_n4' using configuration from collection test_col, trusted=true
   [junit4]   2> 247386 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33874.solr.core.test_col.shard2.replica_n4' (registry 'solr.core.test_col.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 247386 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 247386 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.c.SolrCore [[test_col_shard2_replica_n4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node7/test_col_shard2_replica_n4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node7/./test_col_shard2_replica_n4/data/]
   [junit4]   2> 247388 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8677782416627003]
   [junit4]   2> 247392 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica_n6' using configuration from collection test_col, trusted=true
   [junit4]   2> 247393 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44235.solr.core.test_col.shard2.replica_n6' (registry 'solr.core.test_col.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 247393 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 247393 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.c.SolrCore [[test_col_shard2_replica_n6] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node1/test_col_shard2_replica_n6], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node1/./test_col_shard2_replica_n6/data/]
   [junit4]   2> 247395 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8677782416627003]
   [junit4]   2> 247422 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 247440 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 247440 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 247441 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 247441 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 247442 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 247442 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@6a163c6e[test_col_shard2_replica_n6] main]
   [junit4]   2> 247449 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 247450 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 247451 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 247453 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 247453 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
   [junit4]   2> 247454 INFO  (searcherExecutor-1529-thread-1-processing-n:127.0.0.1:44235_solr x:test_col_shard2_replica_n6 s:shard2 c:test_col r:core_node9) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.c.SolrCore [test_col_shard2_replica_n6] Registered new searcher Searcher@6a163c6e[test_col_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 247454 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592030241993261056
   [junit4]   2> 247455 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml no longer exists.
   [junit4]   2> 247455 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica_n1' using configuration from collection test_col, trusted=true
   [junit4]   2> 247455 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40451.solr.core.test_col.shard1.replica_n1' (registry 'solr.core.test_col.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 247455 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 247455 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.c.SolrCore [[test_col_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node6/test_col_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node6/./test_col_shard1_replica_n1/data/]
   [junit4]   2> 247457 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8677782416627003]
   [junit4]   2> 247505 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 247505 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 247506 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 247507 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 247509 INFO  (qtp1058483725-4613) [n:127.0.0.1:44235_solr c:test_col s:shard2 r:core_node9 x:test_col_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 247510 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 247510 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@5b4c51ba[test_col_shard2_replica_n4] main]
   [junit4]   2> 247514 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 247514 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 247515 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 247515 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 247516 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 0 from ZooKeeper
   [junit4]   2> 247519 INFO  (qtp1556606458-4679) [n:127.0.0.1:33874_solr c:test_col s:shard2 r:core_node7 x:test_col_shard2_replica_n4] o.a.s.s.IndexSchema [test_col_shard2_replica_n4] Schema name=test
   [junit4]   2> 247521 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 247525 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 247525 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 247526 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 247527 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 247529 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 247529 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2fde6342[test_col_shard1_replica_n1] main]
   [junit4]   2> 247530 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
   [junit4]   2> 247531 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
   [junit4]   2> 247531 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 247533 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 0 from ZooKeeper
   [junit4]   2> 247536 INFO  (qtp198913640-4652) [n:127.0.0.1:40451_solr c:test_col s:shard1 r:core_node3 x:test_col_shard1_replica_n1] o.a.s.s.IndexSchema [test_col_shard1_replica_n1] Schema name=test
   [junit4]   2> 247564 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
   [junit4]   2> 247565 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml no longer exists.
   [junit4]   2> 247565 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard3_replica_n10' using configuration from collection test_col, trusted=true
   [junit4]   2> 247566 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58461.solr.core.test_col.shard3.replica_n10' (registry 'solr.core.test_col.shard3.replica_n10') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 247566 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 247566 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.c.SolrCore [[test_col_shard3_replica_n10] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node2/test_col_shard3_replica_n10], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_BE99A5499E4E409F-001/tempDir-001/node2/./test_col_shard3_replica_n10/data/]
   [junit4]   2> 247568 INFO  (qtp706262109-4622) [n:127.0.0.1:58461_solr c:test_col s:shard3 r:core_node12 x:test_col_shard3_replica_n10] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8677782416627003]
   [junit4]   2> 247623 INFO  (zkCallback-632-thread-1-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247623 INFO  (zkCallback-644-thread-2-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247623 INFO  (zkCallback-642-thread-2-processing-n:127.0.0.1:44235_solr) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247623 INFO  (zkCallback-642-thread-1-processing-n:127.0.0.1:44235_solr) [n:127.0.0.1:44235_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247627 INFO  (zkCallback-640-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247629 INFO  (zkCallback-628-thread-2-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247629 INFO  (zkCallback-628-thread-1-processing-n:127.0.0.1:33874_solr) [n:127.0.0.1:33874_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247629 INFO  (zkCallback-632-thread-2-processing-n:127.0.0.1:58461_solr) [n:127.0.0.1:58461_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247629 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
   [junit4]   2> 247629 INFO  (zkCallback-644-thread-1-processing-n:127.0.0.1:53714_solr) [n:127.0.0.1:53714_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247629 INFO  (zkCallback-640-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [7])
   [junit4]   2> 247630 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml no longer exists.
   [junit4]   2> 247630 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard3_replica_n8' using configuration from collection test_col, trusted=true
   [junit4]   2> 247630 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53714.solr.core.test_col.shard3.replica_n8' (registry 'solr.core.test_col.shard3.replica_n8') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@503c9cd8
   [junit4]   2> 247630 INFO  (qtp885682144-4631) [n:127.0.0.1:53714_solr c:test_col s:shard3 r:core_node11 x:test_col_shard3_replica_n8] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 247630

[...truncated too long message...]

  2> 1914088 INFO  (jetty-closer-4630-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:48386_solr
   [junit4]   2> 1914089 INFO  (jetty-closer-4630-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1914090 INFO  (zkCallback-4676-thread-4-processing-n:127.0.0.1:48870_solr) [n:127.0.0.1:48870_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1914090 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006b0c440b000e, likely client has closed socket
   [junit4]   2> 1914091 INFO  (jetty-closer-4630-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@12d18c49{/solr,null,UNAVAILABLE}
   [junit4]   2> 1914091 INFO  (jetty-closer-4630-thread-2) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1914091 INFO  (zkCallback-4666-thread-7-processing-n:127.0.0.1:48386_solr) [n:127.0.0.1:48386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1914093 INFO  (jetty-closer-4630-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1914093 INFO  (zkCallback-4666-thread-9-processing-n:127.0.0.1:48386_solr) [n:127.0.0.1:48386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 1914094 INFO  (jetty-closer-4630-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72175294463279115-127.0.0.1:48386_solr-n_0000000002) closing
   [junit4]   2> 1914094 INFO  (OverseerStateUpdate-72175294463279115-127.0.0.1:48386_solr-n_0000000002) [n:127.0.0.1:48386_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:48386_solr
   [junit4]   2> 1914097 WARN  (OverseerAutoScalingTriggerThread-72175294463279115-127.0.0.1:48386_solr-n_0000000002) [n:127.0.0.1:48386_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1914103 INFO  (jetty-closer-4630-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7923ad51{/solr,null,UNAVAILABLE}
   [junit4]   2> 1914103 INFO  (jetty-closer-4630-thread-1) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 1914105 ERROR (SUITE-TestPullReplica-seed#[BE99A5499E4E409F]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1914106 INFO  (SUITE-TestPullReplica-seed#[BE99A5499E4E409F]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35283 35283
   [junit4]   2> 1914119 INFO  (Thread-3425) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35283 35283
   [junit4]   2> 1914124 WARN  (Thread-3425) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 27 /solr/collections/pull_replica_test_create_delete/terms/shard2
   [junit4]   2> 25 /solr/collections/pull_replica_test_create_delete/terms/shard1
   [junit4]   2> 19 /solr/configs/conf
   [junit4]   2> 17 /solr/collections/pull_replica_test_real_time_get/terms/shard1
   [junit4]   2> 12 /solr/collections/pull_replica_test_add_docs/terms/shard1
   [junit4]   2> 11 /solr/collections/pull_replica_test_remove_all_writer_replicas/terms/shard1
   [junit4]   2> 8 /solr/collections/pull_replica_test_kill_pull_replica/terms/shard1
   [junit4]   2> 8 /solr/collections/pull_replica_test_kill_leader/terms/shard1
   [junit4]   2> 8 /solr/collections/pull_replica_test_add_remove_pull_replica/terms/shard1
   [junit4]   2> 6 /solr/collections/pull_replica_test_add_remove_pull_replica/terms/shard2
   [junit4]   2> 5 /solr/aliases.json
   [junit4]   2> 4 /solr/security.json
   [junit4]   2> 3 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 40 /solr/collections/pull_replica_test_create_delete/state.json
   [junit4]   2> 23 /solr/collections/pull_replica_test_kill_leader/state.json
   [junit4]   2> 23 /solr/collections/pull_replica_test_add_docs/state.json
   [junit4]   2> 22 /solr/collections/pull_replica_test_remove_all_writer_replicas/state.json
   [junit4]   2> 21 /solr/collections/pull_replica_test_kill_pull_replica/state.json
   [junit4]   2> 18 /solr/collections/pull_replica_test_add_remove_pull_replica/state.json
   [junit4]   2> 14 /solr/collections/pull_replica_test_real_time_get/state.json
   [junit4]   2> 5 /solr/clusterprops.json
   [junit4]   2> 5 /solr/clusterstate.json
   [junit4]   2> 3 /solr/autoscaling.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2> 5 /solr/collections
   [junit4]   2> 3 /solr/overseer/queue
   [junit4]   2> 3 /solr/overseer/collection-queue-work
   [junit4]   2> 3 /solr/overseer/queue-work
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplica_BE99A5499E4E409F-001
   [junit4]   2> Feb 10, 2018 4:26:59 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@18bd05ff), locale=el-GR, timezone=America/Nassau
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=234956920,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestPushWriter, TestSQLHandlerNonCloud, TestInitParams, SortSpecParsingTest, TestRTimerTree, TestJsonFacets, TestMacroExpander, TestTrie, TestFieldCacheVsDocValues, BinaryUpdateRequestHandlerTest, TestBlendedInfixSuggestions, SolrGangliaReporterTest, TestSweetSpotSimilarityFactory, TestLeaderElectionZkExpiry, TestSubQueryTransformerDistrib, TestExactStatsCache, DistributedMLTComponentTest, SharedFSAutoReplicaFailoverTest, DocValuesMissingTest, TestDistribDocBasedVersion, ResponseBuilderTest, TestConfigSetImmutable, PingRequestHandlerTest, TestLFUCache, TestNumericTokenStream, CustomCollectionTest, TestRebalanceLeaders, TestDocBasedVersionConstraints, TestUtilizeNode, AnalyticsQueryTest, MultiTermTest, ClassificationUpdateProcessorTest, TestCloudPivotFacet, BufferStoreTest, TestReloadDeadlock, TestMergePolicyConfig, TestSchemaNameResource, TestOverriddenPrefixQueryForCustomFieldType, TestBackupRepositoryFactory, AssignTest, TestClassicSimilarityFactory, TestOnReconnectListenerSupport, TestJmxIntegration, GraphQueryTest, TestImpersonationWithHadoopAuth, TestFieldCacheSort, HdfsDirectoryFactoryTest, TestConfigSetsAPIExclusivity, TestBinaryResponseWriter, TestSchemaSimilarityResource, NodeMutatorTest, TestSimpleQParserPlugin, TestFunctionQuery, RAMDirectoryFactoryTest, SuggesterFSTTest, TestJoin, QueryElevationComponentTest, SolrInfoBeanTest, TestDistribIDF, TestPostingsSolrHighlighter, BlockCacheTest, DistributedFacetSimpleRefinementLongTailTest, TestMultiWordSynonyms, BasicZkTest, DirectSolrConnectionTest, TestPhraseSuggestions, TestConfigsApi, V2ApiIntegrationTest, TestCollationFieldDocValues, ZkNodePropsTest, AlternateDirectoryTest, TestPseudoReturnFields, TestNonDefinedSimilarityFactory, StatsComponentTest, HdfsBasicDistributedZkTest, SolrSlf4jReporterTest, DocumentBuilderTest, MBeansHandlerTest, TestSolrXml, TestRemoteStreaming, ImplicitSnitchTest, TestSolrCloudWithKerberosAlt, SOLR749Test, DistributedFacetPivotWhiteBoxTest, DirectSolrSpellCheckerTest, SuggestComponentTest, TestCloudRecovery, TestSolrJ, TestOrdValues, SolrIndexConfigTest, TestGenericDistributedQueue, OverseerStatusTest, DistanceUnitsTest, ResourceLoaderTest, BlockJoinFacetRandomTest, TestReplicationHandler, DeleteNodeTest, DistributedQueryComponentOptimizationTest, TestPivotHelperCode, OpenExchangeRatesOrgProviderTest, BasicAuthIntegrationTest, TestCustomStream, SpatialFilterTest, SolrMetricReporterTest, OverriddenZkACLAndCredentialsProvidersTest, SubstringBytesRefFilterTest, SchemaApiFailureTest, AutoScalingHandlerTest, TestUnifiedSolrHighlighter, TestCSVResponseWriter, TestInitQParser, ParsingFieldUpdateProcessorsTest, TestSuggestSpellingConverter, CloudMLTQParserTest, HttpPartitionOnCommitTest, DistribCursorPagingTest, TestTolerantSearch, TestExportWriter, ConfigureRecoveryStrategyTest, DirectUpdateHandlerOptimizeTest, TestNamedUpdateProcessors, CoreAdminRequestStatusTest, TestFastOutputStream, FileUtilsTest, TestScoreJoinQPScore, TestZkAclsWithHadoopAuth, HighlighterTest, TestHdfsCloudBackupRestore, SuggesterTSTTest, TestRecovery, TestFiltering, TestZkChroot, AssignBackwardCompatibilityTest, TestElisionMultitermQuery, TestSurroundQueryParser, TestFreeTextSuggestions, PolyFieldTest, TestPerFieldSimilarity, TestFieldSortValues, IndexSchemaRuntimeFieldTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestCharFilters, SpatialRPTFieldTypeTest, ShardSplitTest, TestImplicitCoreProperties, HttpTriggerListenerTest, TestStressLiveNodes, JvmMetricsTest, ManagedSchemaRoundRobinCloudTest, TestCloudManagedSchema, DistributedFacetPivotLargeTest, CdcrUpdateLogTest, TestFieldCollectionResource, TermsComponentTest, HdfsRestartWhileUpdatingTest, DistributedExpandComponentTest, TestNodeAddedTrigger, TestRealTimeGet, ForceLeaderTest, ClassificationUpdateProcessorFactoryTest, TestFoldingMultitermQuery, ConnectionReuseTest, TriLevelCompositeIdRoutingTest, TestPullReplica]
   [junit4] Completed [754/776 (2!)] on J0 in 229.94s, 10 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 48232 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1671 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1671/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

5 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication

Error Message:
Index: 0, Size: 0

Stack Trace:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at __randomizedtesting.SeedInfo.seed([42EEBC021D63B3C6:56A6E7573E640ED8]:0)
        at java.util.ArrayList.rangeCheck(ArrayList.java:657)
        at java.util.ArrayList.get(ArrayList.java:433)
        at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication(TestReplicationHandler.java:561)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 8 in http://127.0.0.1:64181/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 8 in http://127.0.0.1:64181/solr
        at __randomizedtesting.SeedInfo.seed([42EEBC021D63B3C6:831EC5AE30337961]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:885)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:599)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([42EEBC021D63B3C6:69116959871BA616]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTriggerRestoreState(TriggerIntegrationTest.java:368)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([42EEBC021D63B3C6:7350D1E663C9C634]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown(TriggerIntegrationTest.java:1177)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData

Error Message:
Node watch should have fired!

Stack Trace:
java.lang.AssertionError: Node watch should have fired!
        at __randomizedtesting.SeedInfo.seed([42EEBC021D63B3C6:647E535049ED75CC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12331 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 949296 INFO  (SUITE-TestReplicationHandler-seed#[42EEBC021D63B3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/init-core-data-001
   [junit4]   2> 949296 WARN  (SUITE-TestReplicationHandler-seed#[42EEBC021D63B3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 949296 INFO  (SUITE-TestReplicationHandler-seed#[42EEBC021D63B3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 949298 INFO  (SUITE-TestReplicationHandler-seed#[42EEBC021D63B3C6]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 949299 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestIndexFetchWithMasterUrl
   [junit4]   2> 949300 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/collection1
   [junit4]   2> 949302 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:12:37+08:45, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 949303 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 949303 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 949303 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 949303 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@53e14bd4{/solr,null,AVAILABLE}
   [junit4]   2> 949304 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7da4c11c{HTTP/1.1,[http/1.1]}{127.0.0.1:36082}
   [junit4]   2> 949304 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.Server Started @952800ms
   [junit4]   2> 949304 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=36082}
   [junit4]   2> 949304 ERROR (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 949304 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 949304 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 949305 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 949305 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T04:06:47.932Z
   [junit4]   2> 949305 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/solr.xml
   [junit4]   2> 949310 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 949310 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 949311 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 949344 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949364 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949364 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949366 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/.
   [junit4]   2> 949366 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 949366 INFO  (coreLoadExecutor-2454-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 949375 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 949380 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 949382 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 949382 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 949382 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949382 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 949382 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 949382 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=50, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7507546656734022]
   [junit4]   2> 949424 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 949424 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 949424 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=51.0068359375, floorSegmentMB=1.1953125, forceMergeDeletesPctAllowed=10.849206480108275, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5821204590913555
   [junit4]   2> 949425 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a39d6a0[collection1] main]
   [junit4]   2> 949425 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-001/collection1/conf
   [junit4]   2> 949425 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 949425 INFO  (coreLoadExecutor-2454-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 949427 INFO  (searcherExecutor-2455-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3a39d6a0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 949428 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1
   [junit4]   2> 949430 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:12:37+08:45, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 949431 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 949431 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 949431 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 949431 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5da6c0cb{/solr,null,AVAILABLE}
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1500a867{HTTP/1.1,[http/1.1]}{127.0.0.1:58076}
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.Server Started @952928ms
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=58076}
   [junit4]   2> 949432 ERROR (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T04:06:48.059Z
   [junit4]   2> 949432 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/solr.xml
   [junit4]   2> 949436 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 949436 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 949436 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 949469 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949476 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949476 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949478 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/.
   [junit4]   2> 949478 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 949479 INFO  (coreLoadExecutor-2464-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 949488 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 949492 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 949494 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 949494 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 949494 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949494 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 949494 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 949496 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=50, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7507546656734022]
   [junit4]   2> 949536 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 949536 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 949537 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=51.0068359375, floorSegmentMB=1.1953125, forceMergeDeletesPctAllowed=10.849206480108275, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5821204590913555
   [junit4]   2> 949537 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3ef1dfae[collection1] main]
   [junit4]   2> 949538 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1/conf
   [junit4]   2> 949539 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 949539 INFO  (coreLoadExecutor-2464-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 949542 INFO  (qtp699554722-7084) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 949542 INFO  (searcherExecutor-2465-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3ef1dfae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 949542 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 949542 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 949542 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 949542 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 949542 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 949544 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 949544 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7ee60272 commitCommandVersion:0
   [junit4]   2> 949546 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4015ec66[collection1] main]
   [junit4]   2> 949547 INFO  (indexFetcher-2469-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 949547 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1500a867{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 949547 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1019823314
   [junit4]   2> 949547 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 949547 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@151f40a8: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@fe90936
   [junit4]   2> 949547 INFO  (searcherExecutor-2465-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4015ec66[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 949552 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 949552 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e2cca2a: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@64e2373c
   [junit4]   2> 949556 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 949556 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@d0d017a: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4f10760
   [junit4]   2> 949557 INFO  (coreCloseExecutor-2470-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@487a45d7
   [junit4]   2> 949557 INFO  (coreCloseExecutor-2470-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1215972823
   [junit4]   2> 949557 INFO  (coreCloseExecutor-2470-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4a738665: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@1263c0f7
   [junit4]   2> 949565 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5da6c0cb{/solr,null,UNAVAILABLE}
   [junit4]   2> 949565 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 949567 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T06:12:37+08:45, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 949567 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 949567 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4815ce3{/solr,null,AVAILABLE}
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@323ce9f{HTTP/1.1,[http/1.1]}{127.0.0.1:49420}
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.e.j.s.Server Started @953064ms
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=49420}
   [junit4]   2> 949568 ERROR (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 949568 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 949569 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T04:06:48.196Z
   [junit4]   2> 949569 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/solr.xml
   [junit4]   2> 949571 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 949572 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 949572 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 949604 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949611 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949611 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949613 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/.
   [junit4]   2> 949613 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[42EEBC021D63B3C6]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 949613 INFO  (coreLoadExecutor-2477-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 949623 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 949627 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 949629 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 949629 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 949629 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4f5a3bba
   [junit4]   2> 949629 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 949630 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 949631 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=50, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7507546656734022]
   [junit4]   2> 949669 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 949669 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 949669 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=51.0068359375, floorSegmentMB=1.1953125, forceMergeDeletesPctAllowed=10.849206480108275, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5821204590913555
   [junit4]   2> 949670 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c7cd3ee[collection1] main]
   [junit4]   2> 949670 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_42EEBC021D63B3C6-001/solr-instance-002/collection1/conf
   [junit4]   2> 949670 INFO  (coreLoadExecutor-2477-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 949671 INFO  (searcherExecutor-2478-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c7cd3ee[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 949673 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 949675 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 1
   [junit4]   2> 949678 INFO  (qtp851490894-7125) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 949679 INFO  (qtp851490894-7125) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 1
   [junit4]   2> 949680 INFO  (qtp851490894-7128) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 949680 INFO  (qtp851490894-7128) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2c5111ba commitCommandVersion:0
   [junit4]   2> 949682 INFO  (qtp851490894-7128) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@63abe07b[collection1] main]
   [junit4]   2> 949683 INFO  (qtp851490894-7128) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 949684 INFO  (searcherExecutor-2478-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@63abe07b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 949684 INFO  (qtp851490894-7128) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 3
   [junit4]   2> 949685 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 949686 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 949687 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 949688 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 949688 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 949689 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 949689 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 949690 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 949690 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 949690 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 949691 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 949691 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 949692 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 949692 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 949693 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 949693 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 949694 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 949694 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 949695 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 949695 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 949695 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 949696 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 949697 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 949697 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 949697 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 949698 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 949698 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 949698 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 949699 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 949699 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 949700 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 949700 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 949700 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 949701 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 949701 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 949702 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 949702 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 949703 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 949703 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 949704 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 949704 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 949705 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 949705 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 949705 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 949706 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 949706 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 949707 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 949707 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 949708 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 949708 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 949708 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 949709 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 949709 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 949710 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 949710 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 949711 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 949711 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 949711 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 949712 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 949712 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 949713 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 949713 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 949714 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 949714 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 949714 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 949715 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 949715 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 949716 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 949716 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 949717 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 949717 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 949718 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 949718 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 949718 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 949719 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 949719 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 949720 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 949720 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 949721 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 949721 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 949722 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 949722 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 949722 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 949723 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 949723 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 949724 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 949724 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 949724 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 949725 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 949725 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 949726 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 949726 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 949727 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 949727 INFO  (qtp699554722-7093) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 949727 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 949728 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 949728 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 949729 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 949729 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 949730 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 949730 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 949730 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 949731 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 949731 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 949732 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 949732 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 949732 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 949732 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 949733 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 949733 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 949733 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 949734 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 949734 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 949734 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 949735 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 949735 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 949735 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 949736 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 949736 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 949737 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 949737 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 949737 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 949738 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 949738 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 949738 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 949739 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 949739 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 949740 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 949740 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 949740 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 949740 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 949741 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 949741 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 949741 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 949742 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 0
   [junit4]   2> 949742 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 0
   [junit4]   2> 949742 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 949743 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 949743 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 949743 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 949744 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 949744 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 949744 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 949745 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 0
   [junit4]   2> 949745 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 949746 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 949746 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 949747 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 949747 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 949747 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 949748 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
   [junit4]   2> 949748 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 0
   [junit4]   2> 949748 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
   [junit4]   2> 949749 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 0
   [junit4]   2> 949749 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
   [junit4]   2> 949749 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
   [junit4]   2> 949750 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
   [junit4]   2> 949750 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
   [junit4]   2> 949750 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
   [junit4]   2> 949751 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 0
   [junit4]   2> 949751 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
   [junit4]   2> 949752 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
   [junit4]   2> 949752 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
   [junit4]   2> 949752 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
   [junit4]   2> 949753 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
   [junit4]   2> 949753 INFO  (qtp699554722-7094) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
   [junit4]   2> 949753 INFO  (qtp699554722-7087) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 0
   [junit4]   2> 949754 INFO  (qtp699554722-7094) [    x:

[...truncated too long message...]

shard1
   [junit4]   2> 19 /solr/clusterprops.json
   [junit4]   2> 19 /solr/aliases.json
   [junit4]   2> 18 /solr/security.json
   [junit4]   2> 9 /solr/collections/testMetricTrigger/terms/shard1
   [junit4]   2> 9 /solr/configs/conf
   [junit4]   2> 2 /solr/collections/testMetricTrigger/terms/shard2
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 133 /solr/collections/collection1/state.json
   [junit4]   2> 36 /solr/collections/testMetricTrigger/state.json
   [junit4]   2> 19 /solr/clusterstate.json
   [junit4]   2> 6 /solr/autoscaling.json
   [junit4]   2> 3 /solr/collections/collection1/leader_elect/shard1/election/72178290266406950-core_node8-n_0000000003
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406938-127.0.0.1:33356_solr-n_0000000007
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/72178290266406956-core_node6-n_0000000002
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406941-127.0.0.1:57547_solr-n_0000000008
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406950-127.0.0.1:36277_solr-n_0000000011
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406956-127.0.0.1:63332_solr-n_0000000013
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406947-127.0.0.1:62454_solr-n_0000000010
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406929-127.0.0.1:44477_solr-n_0000000004
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406962-127.0.0.1:44032_solr-n_0000000015
   [junit4]   2> 2 /solr/overseer_elect/election/72178290266406917-127.0.0.1:39989_solr-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 19 /solr/live_nodes
   [junit4]   2> 19 /solr/collections
   [junit4]   2> 6 /solr/overseer/queue-work
   [junit4]   2> 6 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 6 /solr/overseer/queue
   [junit4]   2> 6 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/autoscaling/events/node_lost_triggerMR
   [junit4]   2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_42EEBC021D63B3C6-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=Memory), x_s=Lucene50(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=894, maxMBSortInHeap=5.167470593693209, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5555b750), locale=zh-CN, timezone=America/Rainy_River
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=198448152,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [BlockJoinFacetDistribTest, BasicAuthStandaloneTest, TestSearcherReuse, TestScoreJoinQPNoScore, TestCloudNestedDocsSort, UpdateParamsTest, ShowFileRequestHandlerTest, TestComponentsName, SpellPossibilityIteratorTest, TestNumericTerms32, TestQuerySenderListener, ChangedSchemaMergeTest, TestLeaderInitiatedRecoveryThread, SuggestComponentContextFilterQueryTest, TestRandomDVFaceting, HighlighterConfigTest, TestWriterPerf, ZkShardTermsTest, HealthCheckHandlerTest, SolrIndexMetricsTest, NumberUtilsTest, TestSolrCloudWithSecureImpersonation, HdfsRecoveryZkTest, TestLocalFSCloudBackupRestore, QueryResultKeyTest, TestRequestForwarding, TimeRoutedAliasUpdateProcessorTest, DistributedFacetExistsSmallTest, ZkSolrClientTest, IndexBasedSpellCheckerTest, TestSearchPerf, DistribDocExpirationUpdateProcessorTest, LoggingHandlerTest, ConnectionManagerTest, MoveReplicaHDFSTest, DistributedIntervalFacetingTest, ReplaceNodeNoTargetTest, SolrCloudReportersTest, HdfsChaosMonkeyNothingIsSafeTest, TestHashPartitioner, BasicFunctionalityTest, AutoAddReplicasPlanActionTest, TestConfigSetsAPI, TestLuceneMatchVersion, BadCopyFieldTest, DefaultValueUpdateProcessorTest, TestRequestStatusCollectionAPI, AliasIntegrationTest, TestFieldTypeResource, TestCSVLoader, TlogReplayBufferedWhileIndexingTest, TestSolrCoreParser, DisMaxRequestHandlerTest, PrimUtilsTest, HttpPartitionTest, TestFieldCacheReopen, ChaosMonkeyNothingIsSafeTest, TestBulkSchemaConcurrent, TestConfigOverlay, TestCustomSort, RegexBoostProcessorTest, HdfsDirectoryTest, MetricsHandlerTest, TestRestManager, TestMacros, DocValuesTest, TestJettySolrRunner, LeaderFailureAfterFreshStartTest, TestSort, DistributedFacetPivotLongTailTest, TestGeoJSONResponseWriter, TestExactSharedStatsCache, BlockDirectoryTest, TestSystemCollAutoCreate, DistributedQueueTest, TestAddFieldRealTimeGet, FileBasedSpellCheckerTest, TestDistributedMissingSort, TemplateUpdateProcessorTest, TestFastLRUCache, HdfsCollectionsAPIDistributedZkTest, DistributedFacetPivotSmallTest, AtomicUpdatesTest, TestManagedSchemaAPI, TestRecoveryHdfs, TestRandomFaceting, TestLuceneIndexBackCompat, SecurityConfHandlerTest, AnalyticsMergeStrategyTest, ResponseLogComponentTest, TestPerFieldSimilarityWithDefaultOverride, TestCopyFieldCollectionResource, TestCoreAdminApis, TestDocSet, ComputePlanActionTest, TestUseDocValuesAsStored, ScriptEngineTest, AnalysisErrorHandlingTest, RequestHandlersTest, TestRetrieveFieldsOptimizer, TestSchemalessBufferedUpdates, TestSimpleTrackingShardHandler, TestConfigSetsAPIZkFailure, TestNRTOpen, RequestLoggingTest, EchoParamsTest, ExternalFileFieldSortTest, TestSolrCloudWithHadoopAuthPlugin, TestTrieFacet, ConcurrentCreateRoutedAliasTest, HdfsNNFailoverTest, TestFieldCacheSortRandom, MissingSegmentRecoveryTest, HighlighterMaxOffsetTest, TestSolrCoreProperties, ShardRoutingTest, DOMUtilTest, TestDFRSimilarityFactory, TestInPlaceUpdatesDistrib, HdfsSyncSliceTest, TestLRUCache, SolrGraphiteReporterTest, SliceStateTest, TestHashQParserPlugin, SystemLogListenerTest, OutOfBoxZkACLAndCredentialsProvidersTest, RankQueryTest, WrapperMergePolicyFactoryTest, SimpleFacetsTest, TimeZoneUtilsTest, SortByFunctionTest, TestLRUStatsCache, CollectionReloadTest, NodeLostTriggerTest, DeleteStatusTest, DistributedDebugComponentTest, BlobRepositoryCloudTest, TestCrossCoreJoin, TestManagedResource, TestJsonRequest, TestDistributedStatsComponentCardinality, TestClusterStateMutator, ZkControllerTest, UpdateRequestProcessorFactoryTest, TestUpdate, TestSortByMinMaxFunction, TestComputePlanAction, EnumFieldTest, CleanupOldIndexTest, TestHdfsBackupRestoreCore, TestDFISimilarityFactory, TestStressReorder, AsyncCallRequestStatusResponseTest, ClassificationUpdateProcessorIntegrationTest, TestValueSourceCache, TestReqParamsAPI, UtilsToolTest, CdcrBootstrapTest, DocValuesMultiTest, BitVectorTest, TestCodecSupport, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestCloudSchemaless, BadIndexSchemaTest, TestUniqueKeyFieldResource, SystemInfoHandlerTest, TestCloudPseudoReturnFields, TestSolrCloudWithDelegationTokens, DirectUpdateHandlerTest, TestAtomicUpdateErrorCases, CloneFieldUpdateProcessorFactoryTest, TestCollectionsAPIViaSolrCloudCluster, StatsReloadRaceTest, TestHttpShardHandlerFactory, TestCloudSearcherWarming, BlockJoinFacetSimpleTest, ChaosMonkeySafeLeaderWithPullReplicasTest, RegexBytesRefFilterTest, TestMinMaxOnMultiValuedField, TestAuthorizationFramework, TestDocumentBuilder, TestFuzzyAnalyzedSuggestions, DeleteLastCustomShardedReplicaTest, TestDynamicFieldResource, TestLargeCluster, TestStressLucene, ShufflingReplicaListTransformerTest, TestFieldCacheWithThreads, BasicDistributedZkTest, TestDistributedGrouping, TestFaceting, DistributedSpellCheckComponentTest, TestRealTimeGet, TestReload, DistributedTermsComponentTest, TestCoreContainer, TestSolr4Spatial, StatsComponentTest, SolrCmdDistributorTest, TestLazyCores, TestBadConfig, TestIndexSearcher, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTest, SuggesterWFSTTest, PolyFieldTest, NoCacheHeaderTest, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, LukeRequestHandlerTest, TestQueryUtils, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions, TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, IndexSchemaTest, TestJmxIntegration, ReturnFieldsTest, MBeansHandlerTest, QueryParsingTest, TestAnalyzedSuggestions, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, PingRequestHandlerTest, CSVRequestHandlerTest, SearchHandlerTest, TestLFUCache, AlternateDirectoryTest, TestSolrIndexConfig, BadComponentTest, TestStressRecovery, DeleteNodeTest, DeleteShardTest, DocValuesNotIndexedTest, HttpPartitionOnCommitTest, LeaderInitiatedRecoveryOnShardRestartTest, LegacyCloudClusterPropTest, MoveReplicaTest, OverriddenZkACLAndCredentialsProvidersTest, PeerSyncReplicationTest, RemoteQueryErrorTest, ReplaceNodeTest, SSLMigrationTest, SaslZkACLProviderTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestDistribDocBasedVersion, TestMiniSolrCloudClusterSSL, TestPullReplicaErrorHandling, TestRebalanceLeaders, TestSizeLimitedDistributedMap, TestStressInPlaceUpdates, ZkFailoverTest, ConcurrentDeleteAndCreateCollectionTest, SimpleCollectionCreateDeleteTest, TestReplicaProperties, AutoAddReplicasIntegrationTest, MetricTriggerTest, TriggerIntegrationTest]
   [junit4] Completed [600/776 (4!)] on J1 in 197.83s, 14 tests, 2 failures <<< FAILURES!

[...truncated 48943 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1672 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1672/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestUtilizeNode

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestUtilizeNode:     1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestUtilizeNode:
   1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([1393656331314A8C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestUtilizeNode

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([1393656331314A8C]:0)




Build Log:
[...truncated 12664 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestUtilizeNode
   [junit4]   2> 1253285 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/init-core-data-001
   [junit4]   2> 1253285 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1253287 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1253287 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001
   [junit4]   2> 1253287 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1253287 INFO  (Thread-2132) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1253287 INFO  (Thread-2132) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1253289 ERROR (Thread-2132) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1253392 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:33341
   [junit4]   2> 1253394 INFO  (zkConnectionManagerCallback-2059-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253399 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1253399 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1253399 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1253400 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1253400 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1253400 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1253400 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1253401 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1253401 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1253401 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1253402 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1253402 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1253402 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1253403 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1253403 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1253403 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1253403 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29c1ac54{/solr,null,AVAILABLE}
   [junit4]   2> 1253404 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@20b0c291{/solr,null,AVAILABLE}
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@8cc2d46{HTTP/1.1,[http/1.1]}{127.0.0.1:64290}
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.e.j.s.Server Started @1257084ms
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64290}
   [junit4]   2> 1253405 ERROR (jetty-launcher-2056-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@28c4480a{HTTP/1.1,[http/1.1]}{127.0.0.1:56212}
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-1) [    ] o.e.j.s.Server Started @1257085ms
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56212}
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T15:42:01.350Z
   [junit4]   2> 1253405 ERROR (jetty-launcher-2056-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1253405 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1253406 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T15:42:01.351Z
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5cfc921d{/solr,null,AVAILABLE}
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@408ea37e{/solr,null,AVAILABLE}
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@8f0c65d{HTTP/1.1,[http/1.1]}{127.0.0.1:65057}
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.e.j.s.Server Started @1257086ms
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65057}
   [junit4]   2> 1253407 ERROR (jetty-launcher-2056-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1253407 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T15:42:01.352Z
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@711d8e53{HTTP/1.1,[http/1.1]}{127.0.0.1:48165}
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.e.j.s.Server Started @1257087ms
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48165}
   [junit4]   2> 1253408 ERROR (jetty-launcher-2056-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1253408 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T15:42:01.353Z
   [junit4]   2> 1253409 INFO  (zkConnectionManagerCallback-2061-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253409 INFO  (zkConnectionManagerCallback-2063-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253410 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1253410 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1253411 INFO  (zkConnectionManagerCallback-2065-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253411 INFO  (zkConnectionManagerCallback-2067-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253412 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1253418 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1253420 INFO  (jetty-launcher-2056-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33341/solr
   [junit4]   2> 1253421 INFO  (jetty-launcher-2056-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33341/solr
   [junit4]   2> 1253423 INFO  (zkConnectionManagerCallback-2072-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253424 INFO  (zkConnectionManagerCallback-2075-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253426 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100700e9ebf0006, likely client has closed socket
   [junit4]   2> 1253427 INFO  (zkConnectionManagerCallback-2077-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253427 INFO  (jetty-launcher-2056-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33341/solr
   [junit4]   2> 1253429 INFO  (zkConnectionManagerCallback-2080-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253430 INFO  (jetty-launcher-2056-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33341/solr
   [junit4]   2> 1253430 INFO  (zkConnectionManagerCallback-2084-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253433 INFO  (zkConnectionManagerCallback-2087-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253435 INFO  (zkConnectionManagerCallback-2089-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253438 INFO  (zkConnectionManagerCallback-2091-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1253653 INFO  (jetty-launcher-2056-thread-1) [n:127.0.0.1:56212_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1253654 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1253655 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65057_solr
   [junit4]   2> 1253656 INFO  (jetty-launcher-2056-thread-1) [n:127.0.0.1:56212_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56212_solr
   [junit4]   2> 1253658 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.c.Overseer Overseer (id=72180802133098508-127.0.0.1:65057_solr-n_0000000000) starting
   [junit4]   2> 1253659 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1253661 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1253664 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1253668 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1253674 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48165_solr
   [junit4]   2> 1253676 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65057_solr
   [junit4]   2> 1253677 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1253677 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1253678 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1253684 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1253684 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1253684 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1253795 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1253798 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1253800 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64290_solr
   [junit4]   2> 1253808 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1253808 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1253809 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1253822 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1253886 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64290.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253901 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64290.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253901 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64290.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253902 INFO  (jetty-launcher-2056-thread-3) [n:127.0.0.1:64290_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node3/.
   [junit4]   2> 1253915 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48165.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253930 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48165.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253930 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48165.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253931 INFO  (jetty-launcher-2056-thread-4) [n:127.0.0.1:48165_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node4/.
   [junit4]   2> 1253936 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65057.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253946 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65057.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253946 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65057.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253947 INFO  (jetty-launcher-2056-thread-2) [n:127.0.0.1:65057_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node2/.
   [junit4]   2> 1253966 INFO  (jetty-launcher-2056-thread-1) [n:127.0.0.1:56212_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253972 INFO  (jetty-launcher-2056-thread-1) [n:127.0.0.1:56212_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253972 INFO  (jetty-launcher-2056-thread-1) [n:127.0.0.1:56212_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1253973 INFO  (jetty-launcher-2056-thread-1) [n:127.0.0.1:56212_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node1/.
   [junit4]   2> 1254021 INFO  (zkConnectionManagerCallback-2101-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1254025 INFO  (zkConnectionManagerCallback-2105-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1254027 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1254029 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33341/solr ready
   [junit4]   2> 1254038 INFO  (qtp1416522868-9294) [n:127.0.0.1:64290_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1254049 INFO  (qtp1416522868-9294) [n:127.0.0.1:64290_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=10
   [junit4]   2> 1254086 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1254086 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1254093 INFO  (zkConnectionManagerCallback-2108-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1254096 INFO  (qtp1416522868-9295) [n:127.0.0.1:64290_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&name=utilizenodecoll&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1254100 INFO  (OverseerThreadFactory-3188-thread-2-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection utilizenodecoll
   [junit4]   2> 1254100 INFO  (OverseerCollectionConfigSetProcessor-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1254224 INFO  (OverseerStateUpdate-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"utilizenodecoll",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"utilizenodecoll_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:64290/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1254226 INFO  (OverseerStateUpdate-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"utilizenodecoll",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"utilizenodecoll_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:56212/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1254229 INFO  (OverseerStateUpdate-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"utilizenodecoll",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"utilizenodecoll_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:65057/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1254233 INFO  (OverseerStateUpdate-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"utilizenodecoll",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"utilizenodecoll_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:48165/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1254441 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard1_replica_n1&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1254441 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard2_replica_n4&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1254441 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1254441 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1254445 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard1_replica_n2&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1254446 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1254449 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard2_replica_n6&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1254449 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1254562 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254562 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254562 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254562 INFO  (zkCallback-2090-thread-2-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254563 INFO  (zkCallback-2079-thread-2-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254562 INFO  (zkCallback-2076-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254563 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1254563 INFO  (zkCallback-2088-thread-2-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1255483 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1255484 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1255492 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1255500 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1255513 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.s.IndexSchema [utilizenodecoll_shard1_replica_n1] Schema name=cloud-dynamic
   [junit4]   2> 1255514 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.s.IndexSchema [utilizenodecoll_shard2_replica_n6] Schema name=cloud-dynamic
   [junit4]   2> 1255520 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.s.IndexSchema [utilizenodecoll_shard1_replica_n2] Schema name=cloud-dynamic
   [junit4]   2> 1255533 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.s.IndexSchema [utilizenodecoll_shard2_replica_n4] Schema name=cloud-dynamic
   [junit4]   2> 1255611 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1255611 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'utilizenodecoll_shard2_replica_n4' using configuration from collection utilizenodecoll, trusted=true
   [junit4]   2> 1255611 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65057.solr.core.utilizenodecoll.shard2.replica_n4' (registry 'solr.core.utilizenodecoll.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1255611 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1255611 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.SolrCore [[utilizenodecoll_shard2_replica_n4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node2/utilizenodecoll_shard2_replica_n4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node2/./utilizenodecoll_shard2_replica_n4/data/]
   [junit4]   2> 1255616 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1255616 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'utilizenodecoll_shard1_replica_n2' using configuration from collection utilizenodecoll, trusted=true
   [junit4]   2> 1255616 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56212.solr.core.utilizenodecoll.shard1.replica_n2' (registry 'solr.core.utilizenodecoll.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1255616 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1255616 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.SolrCore [[utilizenodecoll_shard1_replica_n2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node1/utilizenodecoll_shard1_replica_n2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node1/./utilizenodecoll_shard1_replica_n2/data/]
   [junit4]   2> 1255628 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1255628 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'utilizenodecoll_shard2_replica_n6' using configuration from collection utilizenodecoll, trusted=true
   [junit4]   2> 1255629 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48165.solr.core.utilizenodecoll.shard2.replica_n6' (registry 'solr.core.utilizenodecoll.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1255629 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1255629 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.c.SolrCore [[utilizenodecoll_shard2_replica_n6] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node4/utilizenodecoll_shard2_replica_n6], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node4/./utilizenodecoll_shard2_replica_n6/data/]
   [junit4]   2> 1255640 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1255640 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'utilizenodecoll_shard1_replica_n1' using configuration from collection utilizenodecoll, trusted=true
   [junit4]   2> 1255641 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64290.solr.core.utilizenodecoll.shard1.replica_n1' (registry 'solr.core.utilizenodecoll.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1255641 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1255641 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.c.SolrCore [[utilizenodecoll_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node3/utilizenodecoll_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node3/./utilizenodecoll_shard1_replica_n1/data/]
   [junit4]   2> 1255937 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1255937 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1255937 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1255937 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1255938 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1255939 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1255938 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1255939 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1255940 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1255941 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1255941 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@7b32371b[utilizenodecoll_shard2_replica_n4] main]
   [junit4]   2> 1255942 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1255942 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1255942 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1255943 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1255944 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4bd379d7[utilizenodecoll_shard1_replica_n1] main]
   [junit4]   2> 1255944 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1255944 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592119760140632064
   [junit4]   2> 1255946 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1255946 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1255947 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1255947 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1255949 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6b13abeb[utilizenodecoll_shard1_replica_n2] main]
   [junit4]   2> 1255950 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@726cbb73[utilizenodecoll_shard2_replica_n6] main]
   [junit4]   2> 1255951 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1255952 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1255952 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1255953 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1255953 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1255954 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592119760151117824
   [junit4]   2> 1255955 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1255956 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592119760153214976
   [junit4]   2> 1255968 INFO  (searcherExecutor-3197-thread-1-processing-n:127.0.0.1:65057_solr x:utilizenodecoll_shard2_replica_n4 s:shard2 c:utilizenodecoll r:core_node7) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.SolrCore [utilizenodecoll_shard2_replica_n4] Registered new searcher Searcher@7b32371b[utilizenodecoll_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1255968 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1255969 INFO  (searcherExecutor-3198-thread-1-processing-n:127.0.0.1:56212_solr x:utilizenodecoll_shard1_replica_n2 s:shard1 c:utilizenodecoll r:core_node5) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.SolrCore [utilizenodecoll_shard1_replica_n2] Registered new searcher Searcher@6b13abeb[utilizenodecoll_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1255970 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1255971 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1255971 INFO  (searcherExecutor-3200-thread-1-processing-n:127.0.0.1:64290_solr x:utilizenodecoll_shard1_replica_n1 s:shard1 c:utilizenodecoll r:core_node3) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.c.SolrCore [utilizenodecoll_shard1_replica_n1] Registered new searcher Searcher@4bd379d7[utilizenodecoll_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1255971 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1255971 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1255976 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592119760174186496
   [junit4]   2> 1255977 INFO  (searcherExecutor-3199-thread-1-processing-n:127.0.0.1:48165_solr x:utilizenodecoll_shard2_replica_n6 s:shard2 c:utilizenodecoll r:core_node8) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.c.SolrCore [utilizenodecoll_shard2_replica_n6] Registered new searcher Searcher@726cbb73[utilizenodecoll_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1256073 INFO  (zkCallback-2076-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256073 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256073 INFO  (zkCallback-2079-thread-2-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256073 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256073 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256073 INFO  (zkCallback-2090-thread-2-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256074 INFO  (zkCallback-2088-thread-2-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256074 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256472 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1256472 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1256472 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56212/solr/utilizenodecoll_shard1_replica_n2/
   [junit4]   2> 1256473 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=utilizenodecoll_shard1_replica_n2 url=http://127.0.0.1:56212/solr START replicas=[http://127.0.0.1:64290/solr/utilizenodecoll_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 1256472 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1256474 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1256474 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:65057/solr/utilizenodecoll_shard2_replica_n4/
   [junit4]   2> 1256475 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=utilizenodecoll_shard2_replica_n4 url=http://127.0.0.1:65057/solr START replicas=[http://127.0.0.1:48165/solr/utilizenodecoll_shard2_replica_n6/] nUpdates=100
   [junit4]   2> 1256478 INFO  (qtp1416522868-9296) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.c.S.Request [utilizenodecoll_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1256480 INFO  (qtp347164769-9315) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.c.S.Request [utilizenodecoll_shard2_replica_n6]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1256775 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=utilizenodecoll_shard1_replica_n2 url=http://127.0.0.1:56212/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1256775 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1256775 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1256775 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1256776 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=utilizenodecoll_shard2_replica_n4 url=http://127.0.0.1:65057/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1256777 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1256777 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1256777 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1256784 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56212/solr/utilizenodecoll_shard1_replica_n2/ shard1
   [junit4]   2> 1256784 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:65057/solr/utilizenodecoll_shard2_replica_n4/ shard2
   [junit4]   2> 1256896 INFO  (zkCallback-2088-thread-2-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256896 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256896 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256896 INFO  (zkCallback-2079-thread-2-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256897 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256897 INFO  (zkCallback-2090-thread-2-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256897 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256897 INFO  (zkCallback-2076-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1256936 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1256936 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1256942 INFO  (qtp1268173969-9292) [n:127.0.0.1:65057_solr c:utilizenodecoll s:shard2 r:core_node7 x:utilizenodecoll_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard2_replica_n4&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2502
   [junit4]   2> 1256943 INFO  (qtp293205333-9302) [n:127.0.0.1:56212_solr c:utilizenodecoll s:shard1 r:core_node5 x:utilizenodecoll_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard1_replica_n2&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2498
   [junit4]   2> 1256975 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr c:utilizenodecoll s:shard1 r:core_node3 x:utilizenodecoll_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard1_replica_n1&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2535
   [junit4]   2> 1256992 INFO  (qtp347164769-9311) [n:127.0.0.1:48165_solr c:utilizenodecoll s:shard2 r:core_node8 x:utilizenodecoll_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=utilizenodecoll_shard2_replica_n6&action=CREATE&numShards=2&collection=utilizenodecoll&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2544
   [junit4]   2> 1257000 INFO  (qtp1416522868-9295) [n:127.0.0.1:64290_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1257098 INFO  (zkCallback-2079-thread-2-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257098 INFO  (zkCallback-2088-thread-2-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257098 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257099 INFO  (zkCallback-2076-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257099 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257099 INFO  (zkCallback-2090-thread-2-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257099 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1257098 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1258001 INFO  (qtp1416522868-9295) [n:127.0.0.1:64290_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf1&name=utilizenodecoll&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=3906
   [junit4]   2> 1258004 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T04:27:37+07:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1258005 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1258005 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1258005 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 1258006 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@700f8831{/solr,null,AVAILABLE}
   [junit4]   2> 1258007 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1bbc684{HTTP/1.1,[http/1.1]}{127.0.0.1:65008}
   [junit4]   2> 1258007 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.e.j.s.Server Started @1261688ms
   [junit4]   2> 1258007 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65008}
   [junit4]   2> 1258007 ERROR (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1258025 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1258025 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1258025 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1258025 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-11T15:42:05.970Z
   [junit4]   2> 1258028 INFO  (zkConnectionManagerCallback-2110-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1258029 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1258097 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33341/solr
   [junit4]   2> 1258099 INFO  (zkConnectionManagerCallback-2114-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1258104 INFO  (zkConnectionManagerCallback-2116-thread-1-processing-n:127.0.0.1:65008_solr) [n:127.0.0.1:65008_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1258106 INFO  (OverseerCollectionConfigSetProcessor-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1258120 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1258132 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1258137 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65008_solr
   [junit4]   2> 1258139 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258139 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258140 INFO  (zkCallback-2104-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258140 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258140 INFO  (zkCallback-2104-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258140 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258141 INFO  (zkCallback-2115-thread-1-processing-n:127.0.0.1:65008_solr) [n:127.0.0.1:65008_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1258309 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65008.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1258318 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65008.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1258318 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65008.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1258320 INFO  (TEST-TestUtilizeNode.test-seed#[1393656331314A8C]) [n:127.0.0.1:65008_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node5/.
   [junit4]   2> 1258484 INFO  (zkConnectionManagerCallback-2120-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1258488 INFO  (qtp347164769-9315) [n:127.0.0.1:48165_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :utilizenode with params node=127.0.0.1:65008_solr&action=UTILIZENODE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1258503 INFO  (qtp347164769-9316) [n:127.0.0.1:48165_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=3
   [junit4]   2> 1258505 INFO  (qtp347164769-9313) [n:127.0.0.1:48165_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258511 INFO  (qtp293205333-9306) [n:127.0.0.1:56212_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=1
   [junit4]   2> 1258512 INFO  (qtp293205333-9304) [n:127.0.0.1:56212_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258514 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1258515 INFO  (qtp1416522868-9300) [n:127.0.0.1:64290_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258520 INFO  (qtp1558180243-9438) [n:127.0.0.1:65008_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=2
   [junit4]   2> 1258524 INFO  (qtp1268173969-9321) [n:127.0.0.1:65057_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard2.replica_n4:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1258526 INFO  (qtp1268173969-9319) [n:127.0.0.1:65057_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258528 INFO  (qtp347164769-9316) [n:127.0.0.1:48165_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard2.replica_n6:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1258529 INFO  (qtp347164769-9313) [n:127.0.0.1:48165_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258530 INFO  (qtp293205333-9306) [n:127.0.0.1:56212_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1258532 INFO  (qtp293205333-9304) [n:127.0.0.1:56212_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258533 INFO  (qtp1416522868-9301) [n:127.0.0.1:64290_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1258534 INFO  (qtp1416522868-9300) [n:127.0.0.1:64290_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258535 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258537 INFO  (qtp1268173969-9323) [n:127.0.0.1:65057_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.utilizenodecoll.shard2.replica_n4:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1258538 INFO  (qtp1268173969-9319) [n:127.0.0.1:65057_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1258540 INFO  (OverseerThreadFactory-3188-thread-3-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 1880011588 sessionWrapper.createTime 1880011587736734, this.sessionWrapper.createTime 1880011587736734
   [junit4]   2> 1258540 INFO  (OverseerThreadFactory-3188-thread-3-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.a.c.MoveReplicaCmd Replica will be moved to node 127.0.0.1:65008_solr: core_node8:{"core":"utilizenodecoll_shard2_replica_n6","base_url":"http://127.0.0.1:48165/solr","node_name":"127.0.0.1:48165_solr","state":"active","type":"NRT"}
   [junit4]   2> 1258542 INFO  (OverseerThreadFactory-3188-thread-3-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:65008_solr for creating new replica
   [junit4]   2> 1258548 INFO  (OverseerStateUpdate-72180802133098508-127.0.0.1:65057_solr-n_0000000000) [n:127.0.0.1:65057_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"utilizenodecoll",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"utilizenodecoll_shard2_replica_n9",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:65008/solr",
   [junit4]   2>   "node_name":"127.0.0.1:65008_solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 1258650 INFO  (zkCallback-2076-thread-1-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258650 INFO  (zkCallback-2076-thread-2-processing-n:127.0.0.1:56212_solr) [n:127.0.0.1:56212_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258651 INFO  (zkCallback-2090-thread-1-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258652 INFO  (zkCallback-2090-thread-2-processing-n:127.0.0.1:65057_solr) [n:127.0.0.1:65057_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258653 INFO  (zkCallback-2079-thread-2-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258653 INFO  (zkCallback-2088-thread-2-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258653 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258655 INFO  (zkCallback-2079-thread-1-processing-n:127.0.0.1:64290_solr) [n:127.0.0.1:64290_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258763 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node10&collection.configName=conf1&name=utilizenodecoll_shard2_replica_n9&action=CREATE&collection=utilizenodecoll&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1258764 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1258779 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1258807 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.s.IndexSchema [utilizenodecoll_shard2_replica_n9] Schema name=cloud-dynamic
   [junit4]   2> 1258853 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1258853 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.c.CoreContainer Creating SolrCore 'utilizenodecoll_shard2_replica_n9' using configuration from collection utilizenodecoll, trusted=true
   [junit4]   2> 1258854 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65008.solr.core.utilizenodecoll.shard2.replica_n9' (registry 'solr.core.utilizenodecoll.shard2.replica_n9') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d1589a7
   [junit4]   2> 1258854 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1258854 INFO  (qtp1558180243-9437) [n:127.0.0.1:65008_solr c:utilizenodecoll s:shard2 r:core_node10 x:utilizenodecoll_shard2_replica_n9] o.a.s.c.SolrCore [[utilizenodecoll_shard2_replica_n9] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node5/utilizenodecoll_shard2_replica_n9], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestUtilizeNode_1393656331314A8C-001/tempDir-001/node5/./utilizenodecoll_shard2_replica_n9/data/]
   [junit4]   2> 1258869 INFO  (zkCallback-2088-thread-1-processing-n:127.0.0.1:48165_solr) [n:127.0.0.1:48165_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/utilizenodecoll/state.json] for collection [utilizenodecoll] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1258869 INFO  (zkCallback-2115-thread-2-proces

[...truncated too long message...]

  2> 17 /solr/collections/utilizenodecoll/terms/shard2
   [junit4]   2> 8 /solr/aliases.json
   [junit4]   2> 8 /solr/clusterprops.json
   [junit4]   2> 6 /solr/security.json
   [junit4]   2> 6 /solr/configs/conf1
   [junit4]   2> 5 /solr/collections/utilizenodecoll/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 61 /solr/collections/utilizenodecoll/state.json
   [junit4]   2> 8 /solr/clusterstate.json
   [junit4]   2> 3 /solr/collections/utilizenodecoll/leader_elect/shard2/election/72180802133098508-core_node7-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72180802133098503-127.0.0.1:56212_solr-n_0000000001
   [junit4]   2> 2 /solr/overseer_elect/election/72180802133098504-127.0.0.1:64290_solr-n_0000000003
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 10 /solr/live_nodes
   [junit4]   2> 9 /solr/collections
   [junit4]   2>
   [junit4]   2> 1296753 INFO  (SUITE-TestUtilizeNode-seed#[1393656331314A8C]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Feb 11, 2018 3:42:44 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Feb 11, 2018 3:43:04 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestUtilizeNode:
   [junit4]   2>    1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]   2>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Feb 11, 2018 3:43:04 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
   [junit4]   2> Feb 11, 2018 3:43:07 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]   2>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]   2>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=623, maxMBSortInHeap=6.19293846178763, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6e59229c), locale=es-NI, timezone=America/Indianapolis
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=2,free=103527064,total=235106304
   [junit4]   2> NOTE: All tests run in this JVM: [ZkCLITest, TestOmitPositions, HLLSerializationTest, TriLevelCompositeIdRoutingTest, SampleTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, DistributedQueryElevationComponentTest, TestBinaryField, FieldMutatingUpdateProcessorTest, TestGenericDistributedQueue, RequiredFieldsTest, TestTriggerIntegration, SolrGangliaReporterTest, SearchRateTriggerTest, TestNamedUpdateProcessors, TestReloadDeadlock, TestRawTransformer, TestBM25SimilarityFactory, TestInitParams, BlockCacheTest, StatsComponentTest, DistanceUnitsTest, TestMaxScoreQueryParser, TestSha256AuthenticationProvider, HdfsBasicDistributedZkTest, CloudExitableDirectoryReaderTest, TolerantUpdateProcessorTest, TestIBSimilarityFactory, TestEmbeddedSolrServerAdminHandler, ForceLeaderTest, TestTolerantUpdateProcessorCloud, HDFSCollectionsAPITest, JavabinLoaderTest, TestNodeAddedTrigger, TestCloudInspectUtil, TermVectorComponentDistributedTest, TestGraphMLResponseWriter, ExecutePlanActionTest, TestClassicSimilarityFactory, ConnectionReuseTest, TestNumericRangeQuery32, DeleteInactiveReplicaTest, TestZkAclsWithHadoopAuth, HdfsRestartWhileUpdatingTest, LeaderInitiatedRecoveryOnShardRestartTest, TestDocBasedVersionConstraints, UninvertDocValuesMergePolicyTest, TestSolrCoreSnapshots, TestSurroundQueryParser, TestDynamicFieldCollectionResource, CollectionTooManyReplicasTest, TestBackupRepositoryFactory, CoreAdminRequestStatusTest, InfoHandlerTest, ImplicitSnitchTest, TestConfigSetImmutable, TestCharFilters, TestInitQParser, BigEndianAscendingWordSerializerTest, StatelessScriptUpdateProcessorFactoryTest, BooleanFieldTest, DistributedVersionInfoTest, TestRTimerTree, TestLegacyFieldReuse, BasicAuthIntegrationTest, TestSQLHandlerNonCloud, ClassificationUpdateProcessorTest, DateFieldTest, TestImpersonationWithHadoopAuth, SearchHandlerTest, PeerSyncWithIndexFingerprintCachingTest, DocExpirationUpdateProcessorFactoryTest, SpatialFilterTest, PropertiesRequestHandlerTest, ZkStateReaderTest, QueryEqualityTest, TestMacroExpander, TestXmlQParser, WordBreakSolrSpellCheckerTest, TestFieldTypeCollectionResource, ReplicaListTransformerTest, SolrShardReporterTest, NodeAddedTriggerTest, QueryElevationComponentTest, TestNestedDocsSort, RecoveryZkTest, RulesTest, TestRandomCollapseQParserPlugin, SchemaApiFailureTest, TestImplicitCoreProperties, TestClusterStateProvider, TestSchemaNameResource, SolrMetricReporterTest, TestPerFieldSimilarity, HdfsDirectoryFactoryTest, LeaderElectionIntegrationTest, DistribJoinFromCollectionTest, TestCSVResponseWriter, NumericFieldsTest, TestPartialUpdateDeduplication, SOLR749Test, DirectSolrSpellCheckerTest, LeaderElectionTest, BasicZkTest, HttpTriggerListenerTest, TestUtilizeNode]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestUtilizeNode -Dtests.seed=1393656331314A8C -Dtests.slow=true -Dtests.locale=es-NI -Dtests.timezone=America/Indianapolis -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestUtilizeNode (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestUtilizeNode:
   [junit4]    >    1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1393656331314A8C]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=9482, name=qtp270146496-9482, state=TIMED_WAITING, group=TGRP-TestUtilizeNode]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
   [junit4]    >         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
   [junit4]    >         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1393656331314A8C]:0)
   [junit4] Completed [216/776 (1!)] on J0 in 66.63s, 1 test, 2 errors <<< FAILURES!

[...truncated 50084 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1673 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1673/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded  at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460)  at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)  at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748) ,time=2}

Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460)
        at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
        at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
,time=2}
        at __randomizedtesting.SeedInfo.seed([79447D1405891A83:F11042CEAB75777B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1191)
        at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1132)
        at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:992)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1019)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12792 lines...]
   [junit4] Suite: org.apache.solr.TestDistributedSearch
   [junit4]   2> 1469724 INFO  (SUITE-TestDistributedSearch-seed#[79447D1405891A83]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/init-core-data-001
   [junit4]   2> 1469725 INFO  (SUITE-TestDistributedSearch-seed#[79447D1405891A83]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1469726 INFO  (SUITE-TestDistributedSearch-seed#[79447D1405891A83]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-9061)
   [junit4]   2> 1469726 INFO  (SUITE-TestDistributedSearch-seed#[79447D1405891A83]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1469975 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/control/cores/collection1
   [junit4]   2> 1469977 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T09:27:37+12:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1469977 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1469978 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1469978 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1469978 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1822d684{/,null,AVAILABLE}
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f617113{HTTP/1.1,[http/1.1]}{127.0.0.1:46742}
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.Server Started @1472934ms
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=46742, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/control/cores}
   [junit4]   2> 1469989 ERROR (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1469989 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-12T04:27:53.394Z
   [junit4]   2> 1469990 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/control/solr.xml
   [junit4]   2> 1469995 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1469995 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1469996 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1470047 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470057 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470057 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470059 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/control/cores
   [junit4]   2> 1470059 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1470060 INFO  (coreLoadExecutor-3331-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1470062 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1470099 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1470118 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1470215 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1470224 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1470225 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470225 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1470225 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/control/cores/collection1/data/]
   [junit4]   2> 1470226 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=91.1806640625, floorSegmentMB=1.037109375, forceMergeDeletesPctAllowed=29.691569415880338, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2749936036211388
   [junit4]   2> 1470276 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1470276 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1470277 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1470277 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1470278 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1780082532, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1470278 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f689e69[collection1] main]
   [junit4]   2> 1470279 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1470284 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1470304 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1470304 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1470304 INFO  (coreLoadExecutor-3331-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592167942019088384
   [junit4]   2> 1470305 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1470305 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1470305 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1470305 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1470305 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1470306 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1470306 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1470306 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1470308 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2f689e69[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1470495 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/shard0/cores/collection1
   [junit4]   2> 1470496 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T09:27:37+12:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1470497 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1470498 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1470498 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1470498 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@73c3324f{/,null,AVAILABLE}
   [junit4]   2> 1470498 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@777a21c8{HTTP/1.1,[http/1.1]}{127.0.0.1:35567}
   [junit4]   2> 1470498 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.Server Started @1473444ms
   [junit4]   2> 1470498 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=35567, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/shard0/cores}
   [junit4]   2> 1470499 ERROR (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1470499 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1470499 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1470499 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1470499 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-12T04:27:53.904Z
   [junit4]   2> 1470499 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/shard0/solr.xml
   [junit4]   2> 1470502 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1470502 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1470503 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1470544 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470552 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470552 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470554 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/shard0/cores
   [junit4]   2> 1470554 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1470555 INFO  (coreLoadExecutor-3342-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1470560 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1470610 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1470622 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1470683 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1470690 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1470690 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75b9286d
   [junit4]   2> 1470690 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1470691 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001/tempDir-001/shard0/cores/collection1/data/]
   [junit4]   2> 1470692 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=91.1806640625, floorSegmentMB=1.037109375, forceMergeDeletesPctAllowed=29.691569415880338, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2749936036211388
   [junit4]   2> 1470762 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1470762 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1470764 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1470764 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1470765 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1780082532, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1470766 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@27255dc3[collection1] main]
   [junit4]   2> 1470766 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1470771 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1470788 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1470789 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1470789 INFO  (coreLoadExecutor-3342-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1592167942527647744
   [junit4]   2> 1470789 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1470789 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1470789 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1470790 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1470790 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1470790 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1470791 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1470791 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1470791 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1470793 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@27255dc3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1470796 INFO  (qtp1013837566-10164) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1592167942532890624)} 0 1
   [junit4]   2> 1470800 INFO  (qtp1043757549-10181) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1592167942536036352)} 0 2
   [junit4]   2> 1470807 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1 (1592167942540230656)]} 0 5
   [junit4]   2> 1470827 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1 (1592167942547570688)]} 0 19
   [junit4]   2> 1470828 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[2 (1592167942567493632)]} 0 0
   [junit4]   2> 1470830 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[2 (1592167942569590784)]} 0 0
   [junit4]   2> 1470831 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[3 (1592167942570639360)]} 0 0
   [junit4]   2> 1470832 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[3 (1592167942572736512)]} 0 0
   [junit4]   2> 1470833 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[4 (1592167942572736512)]} 0 0
   [junit4]   2> 1470833 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[4 (1592167942573785088)]} 0 0
   [junit4]   2> 1470834 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[5 (1592167942574833664)]} 0 0
   [junit4]   2> 1470835 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[5 (1592167942575882240)]} 0 0
   [junit4]   2> 1470837 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[6 (1592167942577979392)]} 0 0
   [junit4]   2> 1470838 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[6 (1592167942579027968)]} 0 0
   [junit4]   2> 1470839 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[7 (1592167942580076544)]} 0 0
   [junit4]   2> 1470840 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[7 (1592167942581125120)]} 0 0
   [junit4]   2> 1470842 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[8 (1592167942583222272)]} 0 0
   [junit4]   2> 1470843 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[8 (1592167942584270848)]} 0 0
   [junit4]   2> 1470843 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[9 (1592167942584270848)]} 0 0
   [junit4]   2> 1470843 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[9 (1592167942584270849)]} 0 0
   [junit4]   2> 1470843 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1592167942584270849,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1470843 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@76883e08 commitCommandVersion:1592167942584270849
   [junit4]   2> 1470855 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@749a891a[collection1] main]
   [junit4]   2> 1470856 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1470857 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@749a891a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c9)))}
   [junit4]   2> 1470857 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 13
   [junit4]   2> 1470858 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1592167942599999488,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1470858 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1bfd33bc commitCommandVersion:1592167942599999488
   [junit4]   2> 1470865 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@d227a95[collection1] main]
   [junit4]   2> 1470866 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1470867 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@d227a95[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c9)))}
   [junit4]   2> 1470867 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 9
   [junit4]   2> 1470869 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[10 (1592167942609436672)]} 0 1
   [junit4]   2> 1470871 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[10 (1592167942611533824)]} 0 1
   [junit4]   2> 1470872 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[11 (1592167942613630976)]} 0 1
   [junit4]   2> 1470874 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[11 (1592167942615728128)]} 0 1
   [junit4]   2> 1470875 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[12 (1592167942616776704)]} 0 0
   [junit4]   2> 1470876 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[12 (1592167942617825280)]} 0 0
   [junit4]   2> 1470876 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[13 (1592167942618873856)]} 0 0
   [junit4]   2> 1470877 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[13 (1592167942619922432)]} 0 0
   [junit4]   2> 1470878 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1001 (1592167942620971008)]} 0 0
   [junit4]   2> 1470879 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1001 (1592167942622019584)]} 0 0
   [junit4]   2> 1470880 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[14 (1592167942623068160)]} 0 0
   [junit4]   2> 1470881 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[14 (1592167942624116736)]} 0 0
   [junit4]   2> 1470882 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[15 (1592167942625165312)]} 0 0
   [junit4]   2> 1470883 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[15 (1592167942626213888)]} 0 0
   [junit4]   2> 1470883 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[16 (1592167942626213888)]} 0 0
   [junit4]   2> 1470884 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[16 (1592167942627262464)]} 0 0
   [junit4]   2> 1470885 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[17 (1592167942628311040)]} 0 0
   [junit4]   2> 1470886 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[17 (1592167942628311040)]} 0 0
   [junit4]   2> 1470886 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[100 (1592167942629359616)]} 0 0
   [junit4]   2> 1470887 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[100 (1592167942630408192)]} 0 0
   [junit4]   2> 1470888 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[101 (1592167942631456768)]} 0 0
   [junit4]   2> 1470888 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[101 (1592167942631456768)]} 0 0
   [junit4]   2> 1470889 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[102 (1592167942632505344)]} 0 0
   [junit4]   2> 1470890 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[102 (1592167942632505344)]} 0 0
   [junit4]   2> 1470890 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[103 (1592167942633553920)]} 0 0
   [junit4]   2> 1470891 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[103 (1592167942634602496)]} 0 0
   [junit4]   2> 1470891 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[104 (1592167942634602496)]} 0 0
   [junit4]   2> 1470892 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[104 (1592167942635651072)]} 0 0
   [junit4]   2> 1470892 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[105 (1592167942635651072)]} 0 0
   [junit4]   2> 1470893 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[105 (1592167942636699648)]} 0 0
   [junit4]   2> 1470893 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[106 (1592167942636699648)]} 0 0
   [junit4]   2> 1470894 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[106 (1592167942637748224)]} 0 0
   [junit4]   2> 1470894 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[107 (1592167942637748224)]} 0 0
   [junit4]   2> 1470895 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[107 (1592167942638796800)]} 0 0
   [junit4]   2> 1470896 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[108 (1592167942638796800)]} 0 0
   [junit4]   2> 1470896 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[108 (1592167942639845376)]} 0 0
   [junit4]   2> 1470897 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[109 (1592167942640893952)]} 0 0
   [junit4]   2> 1470897 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[109 (1592167942640893952)]} 0 0
   [junit4]   2> 1470898 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[110 (1592167942641942528)]} 0 0
   [junit4]   2> 1470899 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[110 (1592167942641942528)]} 0 0
   [junit4]   2> 1470899 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[111 (1592167942642991104)]} 0 0
   [junit4]   2> 1470900 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[111 (1592167942644039680)]} 0 0
   [junit4]   2> 1470900 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[112 (1592167942644039680)]} 0 0
   [junit4]   2> 1470901 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[112 (1592167942645088256)]} 0 0
   [junit4]   2> 1470902 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[113 (1592167942646136832)]} 0 0
   [junit4]   2> 1470903 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[113 (1592167942646136832)]} 0 0
   [junit4]   2> 1470903 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[114 (1592167942647185408)]} 0 0
   [junit4]   2> 1470904 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[114 (1592167942648233984)]} 0 0
   [junit4]   2> 1470905 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[115 (1592167942649282560)]} 0 0
   [junit4]   2> 1470906 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[115 (1592167942650331136)]} 0 0
   [junit4]   2> 1470907 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[116 (1592167942650331136)]} 0 0
   [junit4]   2> 1470908 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[116 (1592167942651379712)]} 0 0
   [junit4]   2> 1470908 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[117 (1592167942652428288)]} 0 0
   [junit4]   2> 1470909 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[117 (1592167942653476864)]} 0 0
   [junit4]   2> 1470910 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[118 (1592167942654525440)]} 0 0
   [junit4]   2> 1470911 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[118 (1592167942654525440)]} 0 0
   [junit4]   2> 1470911 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[119 (1592167942655574016)]} 0 0
   [junit4]   2> 1470912 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[119 (1592167942656622592)]} 0 0
   [junit4]   2> 1470912 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[120 (1592167942656622592)]} 0 0
   [junit4]   2> 1470913 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[120 (1592167942657671168)]} 0 0
   [junit4]   2> 1470913 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[121 (1592167942657671168)]} 0 0
   [junit4]   2> 1470914 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[121 (1592167942658719744)]} 0 0
   [junit4]   2> 1470915 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[122 (1592167942658719744)]} 0 0
   [junit4]   2> 1470915 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[122 (1592167942659768320)]} 0 0
   [junit4]   2> 1470916 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[123 (1592167942660816896)]} 0 0
   [junit4]   2> 1470917 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[123 (1592167942661865472)]} 0 0
   [junit4]   2> 1470917 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[124 (1592167942661865472)]} 0 0
   [junit4]   2> 1470918 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[124 (1592167942662914048)]} 0 0
   [junit4]   2> 1470919 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[125 (1592167942662914048)]} 0 0
   [junit4]   2> 1470919 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[125 (1592167942663962624)]} 0 0
   [junit4]   2> 1470920 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[126 (1592167942663962624)]} 0 0
   [junit4]   2> 1470920 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[126 (1592167942665011200)]} 0 0
   [junit4]   2> 1470921 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[127 (1592167942666059776)]} 0 0
   [junit4]   2> 1470921 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[127 (1592167942666059776)]} 0 0
   [junit4]   2> 1470922 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[128 (1592167942667108352)]} 0 0
   [junit4]   2> 1470922 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[128 (1592167942667108352)]} 0 0
   [junit4]   2> 1470923 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[129 (1592167942668156928)]} 0 0
   [junit4]   2> 1470924 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[129 (1592167942669205504)]} 0 0
   [junit4]   2> 1470925 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[130 (1592167942669205504)]} 0 0
   [junit4]   2> 1470925 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[130 (1592167942670254080)]} 0 0
   [junit4]   2> 1470926 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[131 (1592167942671302656)]} 0 0
   [junit4]   2> 1470926 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[131 (1592167942671302656)]} 0 0
   [junit4]   2> 1470927 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[132 (1592167942672351232)]} 0 0
   [junit4]   2> 1470928 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[132 (1592167942672351232)]} 0 0
   [junit4]   2> 1470928 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[133 (1592167942673399808)]} 0 0
   [junit4]   2> 1470929 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[133 (1592167942674448384)]} 0 0
   [junit4]   2> 1470930 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[134 (1592167942674448384)]} 0 0
   [junit4]   2> 1470930 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[134 (1592167942675496960)]} 0 0
   [junit4]   2> 1470931 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[135 (1592167942676545536)]} 0 0
   [junit4]   2> 1470932 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[135 (1592167942676545536)]} 0 0
   [junit4]   2> 1470932 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[136 (1592167942677594112)]} 0 0
   [junit4]   2> 1470933 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[136 (1592167942678642688)]} 0 0
   [junit4]   2> 1470933 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[137 (1592167942678642688)]} 0 0
   [junit4]   2> 1470934 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[137 (1592167942679691264)]} 0 0
   [junit4]   2> 1470935 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[138 (1592167942680739840)]} 0 0
   [junit4]   2> 1470936 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[138 (1592167942680739840)]} 0 0
   [junit4]   2> 1470936 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[139 (1592167942681788416)]} 0 0
   [junit4]   2> 1470937 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[139 (1592167942682836992)]} 0 0
   [junit4]   2> 1470938 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[140 (1592167942683885568)]} 0 0
   [junit4]   2> 1470939 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[140 (1592167942683885568)]} 0 0
   [junit4]   2> 1470939 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[141 (1592167942684934144)]} 0 0
   [junit4]   2> 1470940 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[141 (1592167942685982720)]} 0 0
   [junit4]   2> 1470941 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[142 (1592167942685982720)]} 0 0
   [junit4]   2> 1470941 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[142 (1592167942687031296)]} 0 0
   [junit4]   2> 1470942 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[143 (1592167942688079872)]} 0 0
   [junit4]   2> 1470943 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[143 (1592167942688079872)]} 0 0
   [junit4]   2> 1470943 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[144 (1592167942689128448)]} 0 0
   [junit4]   2> 1470944 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[144 (1592167942690177024)]} 0 0
   [junit4]   2> 1470944 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[145 (1592167942690177024)]} 0 0
   [junit4]   2> 1470945 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[145 (1592167942691225600)]} 0 0
   [junit4]   2> 1470945 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[146 (1592167942691225600)]} 0 0
   [junit4]   2> 1470946 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[146 (1592167942692274176)]} 0 0
   [junit4]   2> 1470947 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[147 (1592167942692274176)]} 0 0
   [junit4]   2> 1470947 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[147 (1592167942693322752)]} 0 0
   [junit4]   2> 1470948 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[148 (1592167942693322752)]} 0 0
   [junit4]   2> 1470948 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[148 (1592167942694371328)]} 0 0
   [junit4]   2> 1470949 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[149 (1592167942695419904)]} 0 0
   [junit4]   2> 1470950 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[149 (1592167942696468480)]} 0 0
   [junit4]   2> 1470950 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1592167942696468480,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1470950 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@76883e08 commitCommandVersion:1592167942696468480
   [junit4]   2> 1470960 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@41fe23ba[collection1] main]
   [junit4]   2> 1470960 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1470962 INFO  (searcherExecutor-3332-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@41fe23ba[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c9) Uninverting(_1(8.0.0):c59)))}
   [junit4]   2> 1470962 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 12
   [junit4]   2> 1470963 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1592167942710099968,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1470963 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1bfd33bc commitCommandVersion:1592167942710099968
   [junit4]   2> 1470970 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@65b7d242[collection1] main]
   [junit4]   2> 1470971 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1470973 INFO  (searcherExecutor-3343-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@65b7d242[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c9) Uninverting(_1(8.0.0):c59)))}
   [junit4]   2> 1470973 INFO  (qtp1043757549-10187) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 9
   [junit4]   2> 1470974 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.h.RequestHandlerBase Ignoring exception matching parameter cannot be negative
   [junit4]   2> 1470974 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*&distrib=false&start=-1&wt=javabin&version=2} status=400 QTime=0
   [junit4]   2> 1470975 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.h.RequestHandlerBase Ignoring exception matching parameter cannot be negative
   [junit4]   2> 1470975 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*&distrib=false&start=0&rows=-1&wt=javabin&version=2} status=400 QTime=0
   [junit4]   2> 1470977 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1470983 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674385&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1470986 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674385&ids=11,100,101,113,126,104,1001,141,131,10&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1470995 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=14
   [junit4]   2> 1470996 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_ti1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471012 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+asc&shard.url=[ff01::114]:33332/|[ff01::213]:33332/|[ff01::083]:33332/|http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674402&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471014 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674402&ids=133,111,145,2,102,15,138,139,8,118&isShard=true&shard.url=[ff01::114]:33332/|[ff01::213]:33332/|[ff01::083]:33332/|http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471021 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::114]:33332/|[ff01::213]:33332/|[ff01::083]:33332/|http://127.0.0.1:35567//collection1&sort=n_ti1+asc&wt=javabin&version=2} hits=68 status=0 QTime=23
   [junit4]   2> 1471022 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_f1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471024 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_f1+desc&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674429&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471026 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674429&ids=121,144,111,1,134,5,17,128,9,140&isShard=true&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471033 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|http://127.0.0.1:35567//collection1&sort=n_f1+desc&wt=javabin&version=2} hits=68 status=0 QTime=9
   [junit4]   2> 1471034 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_f1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471035 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_f1+asc&shard.url=http://127.0.0.1:35567//collection1|[ff01::083]:33332/|[ff01::114]:33332/&rows=10&version=2&q=*:*&NOW=1518409674440&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471040 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674440&ids=11,123,13,101,14,113,124,147,105,149&isShard=true&shard.url=http://127.0.0.1:35567//collection1|[ff01::083]:33332/|[ff01::114]:33332/&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 1471047 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1|[ff01::083]:33332/|[ff01::114]:33332/&sort=n_f1+asc&wt=javabin&version=2} hits=68 status=0 QTime=12
   [junit4]   2> 1471049 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_tf1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471051 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tf1+desc&shard.url=[ff01::114]:33332/|http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674455&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471052 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674455&ids=132,100,12,133,147,125,17,127,117,120&isShard=true&shard.url=[ff01::114]:33332/|http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471055 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::114]:33332/|http://127.0.0.1:35567//collection1&sort=n_tf1+desc&wt=javabin&version=2} hits=68 status=0 QTime=5
   [junit4]   2> 1471056 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_tf1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471058 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tf1+asc&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1518409674462&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471059 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674462&ids=145,113,114,128,7,118,129,9,108,140&isShard=true&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1|[ff01::114]:33332/|[ff01::213]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471060 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|http://127.0.0.1:35567//collection1|[ff01::114]:33332/|[ff01::213]:33332/&sort=n_tf1+asc&wt=javabin&version=2} hits=68 status=0 QTime=3
   [junit4]   2> 1471061 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_d1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471063 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_d1+desc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674467&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471064 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674467&ids=143,101,114,136,139,8,129,107,130,142&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471065 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=n_d1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471065 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_d1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471067 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_d1+asc&shard.url=http://127.0.0.1:35567//collection1|[ff01::114]:33332/|[ff01::083]:33332/&rows=10&version=2&q=*:*&NOW=1518409674471&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471069 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674471&ids=110,112,134,2,5,16,106,109,140,131&isShard=true&shard.url=http://127.0.0.1:35567//collection1|[ff01::114]:33332/|[ff01::083]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471070 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1|[ff01::114]:33332/|[ff01::083]:33332/&sort=n_d1+asc&wt=javabin&version=2} hits=68 status=0 QTime=3
   [junit4]   2> 1471071 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_td1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471072 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_td1+desc&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1|[ff01::114]:33332/&rows=10&version=2&q=*:*&NOW=1518409674476&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471074 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674476&ids=132,12,2,3,115,127,128,7,109,130&isShard=true&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1|[ff01::114]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471075 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|http://127.0.0.1:35567//collection1|[ff01::114]:33332/&sort=n_td1+desc&wt=javabin&version=2} hits=68 status=0 QTime=3
   [junit4]   2> 1471075 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_td1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471077 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_td1+asc&shard.url=http://127.0.0.1:35567//collection1|[ff01::213]:33332/|[ff01::114]:33332/&rows=10&version=2&q=*:*&NOW=1518409674482&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471078 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674482&ids=123,145,134,102,17,138,107,9,141,120&isShard=true&shard.url=http://127.0.0.1:35567//collection1|[ff01::213]:33332/|[ff01::114]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471079 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1|[ff01::213]:33332/|[ff01::114]:33332/&sort=n_td1+asc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471080 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_l1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471081 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_l1+desc&shard.url=http://127.0.0.1:35567//collection1|[ff01::114]:33332/&rows=10&version=2&q=*:*&NOW=1518409674486&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471082 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674486&ids=110,3,135,137,117,7,139,9,108,142&isShard=true&shard.url=http://127.0.0.1:35567//collection1|[ff01::114]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471083 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1|[ff01::114]:33332/&sort=n_l1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471083 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_l1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471085 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_l1+asc&shard.url=http://127.0.0.1:35567//collection1|[ff01::114]:33332/&rows=10&version=2&q=*:*&NOW=1518409674489&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471086 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674489&ids=144,13,146,114,5,128,1001,109,130,10&isShard=true&shard.url=http://127.0.0.1:35567//collection1|[ff01::114]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471087 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1|[ff01::114]:33332/&sort=n_l1+asc&wt=javabin&version=2} hits=68 status=0 QTime=3
   [junit4]   2> 1471088 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_tl1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471090 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tl1+desc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674495&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471092 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674495&ids=143,100,112,146,114,15,104,17,130,120&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471092 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=n_tl1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471093 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_tl1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471094 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tl1+asc&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674499&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471095 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674499&ids=110,145,113,135,4,126,149,128,119,109&isShard=true&shard.url=[ff01::083]:33332/|http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471096 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|http://127.0.0.1:35567//collection1&sort=n_tl1+asc&wt=javabin&version=2} hits=68 status=0 QTime=1
   [junit4]   2> 1471096 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_dt1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471098 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_dt1+desc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674502&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471098 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674502&ids=133,100,12,101,3,14,137,104,130,10&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471099 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=n_dt1+desc&wt=javabin&version=2} hits=68 status=0 QTime=1
   [junit4]   2> 1471100 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_dt1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471101 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_dt1+asc&shard.url=[ff01::213]:33332/|http://127.0.0.1:35567//collection1|[ff01::083]:33332/&rows=10&version=2&q=*:*&NOW=1518409674505&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471102 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674505&ids=144,2,124,146,125,136,148,117,109,140&isShard=true&shard.url=[ff01::213]:33332/|http://127.0.0.1:35567//collection1|[ff01::083]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471102 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=[ff01::213]:33332/|http://127.0.0.1:35567//collection1|[ff01::083]:33332/&sort=n_dt1+asc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471103 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_tdt1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471105 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tdt1+desc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674509&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471105 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674509&ids=2,102,124,146,14,114,126,129,107,119&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471106 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=n_tdt1+desc&wt=javabin&version=2} hits=68 status=0 QTime=1
   [junit4]   2> 1471107 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_tdt1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471108 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tdt1+asc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674512&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471109 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674512&ids=3,135,136,15,16,148,138,105,149,108&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471110 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=n_tdt1+asc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471111 INFO  (qtp1013837566-10166) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=a_i1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1471113 INFO  (qtp1043757549-10188) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=a_i1+desc&shard.url=http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674517&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1471114 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674517&ids=1,12,2,13,3,5,7,8,9,10&isShard=true&shard.url=http://127.0.0.1:35567//collection1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1471115 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:35567//collection1&sort=a_i1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
   [junit4]   2> 1471119 INFO  (qtp1013837566-10168) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort={!func}testfunc(add(a_i1,5))+desc&wt=javabin&version=2} hits=68 status=0 QTime=3
   [junit4]   2> 1471124 INFO  (qtp1043757549-10184) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort={!func}testfunc(add(a_i1,5))+desc&shard.url=[ff01::213]:33332/|http://127.0.0.1:35567//collection1&rows=10&version=2&q=*:*&NOW=1518409674524&isShard=true&wt=javabin} hits=68 status=0 QTime=4
   [junit4]   2> 1471125 INFO  (qtp1043757549-10185) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1518409674524&ids=1,12,2,13,3,5,7,

[...truncated too long message...]

   2> 1497908 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1617021645
   [junit4]   2> 1497908 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1497908 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4ce20dc7: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@607c5783
   [junit4]   2> 1497916 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1497916 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@147de26d: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2d099855
   [junit4]   2> 1497920 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1497920 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@193e3820: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2936092c
   [junit4]   2> 1497922 INFO  (coreCloseExecutor-3389-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@15e046b2
   [junit4]   2> 1497922 INFO  (coreCloseExecutor-3389-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=367019698
   [junit4]   2> 1497922 INFO  (coreCloseExecutor-3389-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4ba54f58: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@735722c2
   [junit4]   2> 1497932 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6c745e1{/,null,UNAVAILABLE}
   [junit4]   2> 1497932 INFO  (TEST-TestDistributedSearch.test-seed#[79447D1405891A83]) [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistributedSearch -Dtests.method=test -Dtests.seed=79447D1405891A83 -Dtests.slow=true -Dtests.locale=fi-FI -Dtests.timezone=America/Rio_Branco -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 28.2s J1 | TestDistributedSearch.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460)
   [junit4]    > at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
   [junit4]    > at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]    > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > ,time=2}
   [junit4]    > at __randomizedtesting.SeedInfo.seed([79447D1405891A83:F11042CEAB75777B]:0)
   [junit4]    > at org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1191)
   [junit4]    > at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1132)
   [junit4]    > at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:992)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1019)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.TestDistributedSearch_79447D1405891A83-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {severity=PostingsFormat(name=MockRandom), foo_sev_enum=PostingsFormat(name=Direct), SubjectTerms_mfacet=PostingsFormat(name=Memory), multiDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), rnd_b=PostingsFormat(name=Direct), a_s=PostingsFormat(name=MockRandom), oddField_s=PostingsFormat(name=MockRandom), lowerfilt1and2=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=PostingsFormat(name=Memory), foo_b=PostingsFormat(name=MockRandom), lowerfilt=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=PostingsFormat(name=Direct), text=PostingsFormat(name=MockRandom)}, docValues:{other_tl1=DocValuesFormat(name=Lucene70), range_facet_l_dv=DocValuesFormat(name=Direct), n_l1=DocValuesFormat(name=Lucene70), a_n_tdt=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Lucene70), n_td1=DocValuesFormat(name=Lucene70), n_d1=DocValuesFormat(name=Lucene70), range_facet_l=DocValuesFormat(name=Lucene70), n_f1=DocValuesFormat(name=Lucene70), n_tl1=DocValuesFormat(name=Lucene70), n_tf1=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Lucene70), severity=DocValuesFormat(name=Asserting), foo_sev_enum=DocValuesFormat(name=Direct), n_dt1=DocValuesFormat(name=Lucene70), a_i1=DocValuesFormat(name=Lucene70), b_n_tdt=DocValuesFormat(name=Lucene70), n_ti1=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Lucene70), n_tdt1=DocValuesFormat(name=Lucene70), id_i1=DocValuesFormat(name=Lucene70), foo_d=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Lucene70), foo_f=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1178, maxMBSortInHeap=6.312989121396517, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@186f87a6), locale=fi-FI, timezone=America/Rio_Branco
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=54772656,total=393138176
   [junit4]   2> NOTE: All tests run in this JVM: [EchoParamsTest, TestDefaultStatsCache, InfixSuggestersTest, TestFieldCacheWithThreads, LargeFieldTest, TermVectorComponentTest, DeleteLastCustomShardedReplicaTest, TestScoreJoinQPNoScore, LoggingHandlerTest, BaseCdcrDistributedZkTest, ClusterStateTest, HdfsNNFailoverTest, TestManagedSchemaAPI, TestCrossCoreJoin, TestSolrCLIRunExample, DistributedFacetPivotLongTailTest, CurrencyRangeFacetCloudTest, MoveReplicaHDFSTest, MetricUtilsTest, SearchHandlerTest, V2StandaloneTest, SaslZkACLProviderTest, TestEmbeddedSolrServerConstructors, TestTlogReplica, DocValuesMultiTest, SolrCmdDistributorTest, TestAnalyzeInfixSuggestions, TestAuthorizationFramework, CdcrReplicationHandlerTest, PreAnalyzedFieldManagedSchemaCloudTest, TestJettySolrRunner, OverseerTest, TestLeaderInitiatedRecoveryThread, TestHighFrequencyDictionaryFactory, BigEndianAscendingWordDeserializerTest, ClusterStateUpdateTest, CreateRoutedAliasTest, SystemInfoHandlerTest, CollectionReloadTest, NoCacheHeaderTest, TestSubQueryTransformerCrossCore, StatsReloadRaceTest, TestShardHandlerFactory, HighlighterConfigTest, WrapperMergePolicyFactoryTest, SmileWriterTest, TestCollectionAPIs, BasicDistributedZk2Test, DistributedIntervalFacetingTest, MetricTriggerTest, HdfsWriteToMultipleCollectionsTest, TestHalfAndHalfDocValues, DeleteReplicaTest, BadIndexSchemaTest, TestSystemCollAutoCreate, MetricsConfigTest, TestHttpShardHandlerFactory, TestHdfsBackupRestoreCore, TestStressInPlaceUpdates, CSVRequestHandlerTest, NumberUtilsTest, SoftAutoCommitTest, TestCustomDocTransformer, CoreAdminCreateDiscoverTest, TestTrieFacet, SpellPossibilityIteratorTest, TestLegacyField, TestBulkSchemaConcurrent, TestInfoStreamLogging, TestRuleBasedAuthorizationPlugin, TestDynamicFieldResource, FileBasedSpellCheckerTest, TestConfigSetsAPIZkFailure, HdfsCollectionsAPIDistributedZkTest, SpellCheckCollatorTest, TestHashQParserPlugin, TestFastLRUCache, TestSchemalessBufferedUpdates, TestLegacyNumericUtils, TestPrepRecovery, SolrCoreCheckLockOnStartupTest, TriggerIntegrationTest, TestExceedMaxTermLength, TestChildDocTransformer, TestFuzzyAnalyzedSuggestions, TestJsonRequest, BlockDirectoryTest, TestHashPartitioner, TestStressLucene, TestDistributedSearch]
   [junit4] Completed [242/776 (1!)] on J1 in 28.23s, 1 test, 1 failure <<< FAILURES!

[...truncated 50250 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1674 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1674/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A4833FA978AF33C2:1E8F08262747E58D]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.junit.Assert.assertNull(Assert.java:562)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1575)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData

Error Message:
Node watch should have fired!

Stack Trace:
java.lang.AssertionError: Node watch should have fired!
        at __randomizedtesting.SeedInfo.seed([A4833FA978AF33C2:8213D0FB2C21F5C8]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.sql.JdbcTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.sql.JdbcTest:     1) Thread[id=2287, name=qtp97882992-2287, state=TIMED_WAITING, group=TGRP-JdbcTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.sql.JdbcTest:
   1) Thread[id=2287, name=qtp97882992-2287, state=TIMED_WAITING, group=TGRP-JdbcTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([9E51983DB29857F7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.sql.JdbcTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2287, name=qtp97882992-2287, state=TIMED_WAITING, group=TGRP-JdbcTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2287, name=qtp97882992-2287, state=TIMED_WAITING, group=TGRP-JdbcTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([9E51983DB29857F7]:0)




Build Log:
[...truncated 12275 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestDistribStateManager_A4833FA978AF33C2-001/init-core-data-001
   [junit4]   2> 610295 WARN  (SUITE-TestDistribStateManager-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 610296 INFO  (SUITE-TestDistribStateManager-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 610297 INFO  (SUITE-TestDistribStateManager-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 610300 INFO  (SUITE-TestDistribStateManager-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 610302 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateMode
   [junit4]   2> 610306 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
   [junit4]   2> 610308 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
   [junit4]   2> 610308 INFO  (TEST-TestDistribStateManager.testCreateMode-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateMode
   [junit4]   2> 610309 INFO  (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testListData
   [junit4]   2> 610309 INFO  (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 610310 INFO  (Thread-702) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 610310 INFO  (Thread-702) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 610312 ERROR (Thread-702) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 610409 INFO  (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer start zk server on port:43488
   [junit4]   2> 610413 INFO  (zkConnectionManagerCallback-837-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 610414 INFO  (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 610422 ERROR (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 610422 INFO  (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43488 43488
   [junit4]   2> 615562 INFO  (Thread-702) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43488 43488
   [junit4]   2> 615566 INFO  (TEST-TestDistribStateManager.testListData-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testListData
   [junit4]   2> 615568 INFO  (TEST-TestDistribStateManager.testHasData-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testHasData
   [junit4]   2> 615568 INFO  (TEST-TestDistribStateManager.testHasData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
   [junit4]   2> 615568 INFO  (TEST-TestDistribStateManager.testHasData-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testHasData
   [junit4]   2> 615569 INFO  (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMulti
   [junit4]   2> 615570 INFO  (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 615570 INFO  (Thread-703) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 615570 INFO  (Thread-703) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 615584 ERROR (Thread-703) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 615670 INFO  (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer start zk server on port:41858
   [junit4]   2> 615686 INFO  (zkConnectionManagerCallback-839-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 615686 INFO  (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 615691 ERROR (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 615693 INFO  (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41858 41858
   [junit4]   2> 621573 INFO  (Thread-703) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41858 41858
   [junit4]   2> 621574 INFO  (TEST-TestDistribStateManager.testMulti-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMulti
   [junit4]   2> 621576 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testGetSetRemoveData
   [junit4]   2> 621576 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 621576 INFO  (Thread-704) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 621576 INFO  (Thread-704) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 621578 ERROR (Thread-704) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 621680 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer start zk server on port:52655
   [junit4]   2> 621683 INFO  (zkConnectionManagerCallback-841-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 621683 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 621694 INFO  (zkConnectionManagerCallback-843-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 621694 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
   [junit4]   2> 626696 ERROR (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 626697 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52655 52655
   [junit4]   2> 636752 INFO  (Thread-704) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52655 52655
   [junit4]   2> 636753 INFO  (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testGetSetRemoveData
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistribStateManager -Dtests.method=testGetSetRemoveData -Dtests.seed=A4833FA978AF33C2 -Dtests.slow=true -Dtests.locale=fr-FR -Dtests.timezone=Europe/Budapest -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 15.2s J0 | TestDistribStateManager.testGetSetRemoveData <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Node watch should have fired!
   [junit4]    > at __randomizedtesting.SeedInfo.seed([A4833FA978AF33C2:8213D0FB2C21F5C8]:0)
   [junit4]    > at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestDistribStateManager_A4833FA978AF33C2-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=267, maxMBSortInHeap=6.588898369229349, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5dc13d58), locale=fr-FR, timezone=Europe/Budapest
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=80739192,total=185090048
   [junit4]   2> NOTE: All tests run in this JVM: [BufferStoreTest, TestRealTimeGet, SolrTestCaseJ4Test, TestEmbeddedSolrServerAdminHandler, TestBinaryField, InfoHandlerTest, TestRestoreCore, TestCloudRecovery, RequiredFieldsTest, DistribCursorPagingTest, DirectoryFactoryTest, DocValuesMissingTest, TestCloudPivotFacet, PeerSyncReplicationTest, TestPayloadScoreQParserPlugin, DateMathParserTest, SuggesterTSTTest, DistributedExpandComponentTest, TestSolr4Spatial2, CopyFieldTest, PropertiesRequestHandlerTest, TestReloadDeadlock, TestSweetSpotSimilarityFactory, CustomHighlightComponentTest, SolrCoreMetricManagerTest, CollectionsAPIDistributedZkTest, TestFieldSortValues, CollectionTooManyReplicasTest, TestJmxIntegration, TestConfigSetImmutable, ForceLeaderTest, CustomCollectionTest, TestNumericTerms64, TestComplexPhraseQParserPlugin, DistributedFacetSimpleRefinementLongTailTest, TestDistribStateManager]
   [junit4] Completed [81/776 (1!)] on J0 in 26.47s, 5 tests, 1 failure <<< FAILURES!

[...truncated 1755 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A4833FA978AF33C2-001/init-core-data-001
   [junit4]   2> 4381132 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4381133 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 4381133 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 4381133 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A4833FA978AF33C2-001/tempDir-001
   [junit4]   2> 4381133 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4381134 INFO  (Thread-6326) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4381134 INFO  (Thread-6326) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4381137 ERROR (Thread-6326) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4381234 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:59561
   [junit4]   2> 4381240 INFO  (zkConnectionManagerCallback-7736-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381250 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4381250 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4381252 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4381252 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4381252 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 4381252 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4381252 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4381252 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@f0ce39{/solr,null,AVAILABLE}
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3f0068dd{/solr,null,AVAILABLE}
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74f12367{HTTP/1.1,[http/1.1]}{127.0.0.1:65526}
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-2) [    ] o.e.j.s.Server Started @4389520ms
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65526}
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@366bbf54{HTTP/1.1,[http/1.1]}{127.0.0.1:54692}
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-1) [    ] o.e.j.s.Server Started @4389520ms
   [junit4]   2> 4381253 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54692}
   [junit4]   2> 4381253 ERROR (jetty-launcher-7733-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4381254 ERROR (jetty-launcher-7733-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-13T00:09:47.591Z
   [junit4]   2> 4381254 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-13T00:09:47.591Z
   [junit4]   2> 4381256 INFO  (zkConnectionManagerCallback-7740-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381257 INFO  (zkConnectionManagerCallback-7738-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381257 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4381257 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4381264 INFO  (jetty-launcher-7733-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59561/solr
   [junit4]   2> 4381264 INFO  (jetty-launcher-7733-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59561/solr
   [junit4]   2> 4381266 INFO  (zkConnectionManagerCallback-7748-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381266 INFO  (zkConnectionManagerCallback-7746-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381269 INFO  (zkConnectionManagerCallback-7750-thread-1-processing-n:127.0.0.1:54692_solr) [n:127.0.0.1:54692_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381269 INFO  (zkConnectionManagerCallback-7752-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381398 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4381399 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65526_solr
   [junit4]   2> 4381400 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.Overseer Overseer (id=72188471013539846-127.0.0.1:65526_solr-n_0000000000) starting
   [junit4]   2> 4381402 INFO  (jetty-launcher-7733-thread-1) [n:127.0.0.1:54692_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4381403 INFO  (jetty-launcher-7733-thread-1) [n:127.0.0.1:54692_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54692_solr
   [junit4]   2> 4381411 INFO  (zkCallback-7749-thread-1-processing-n:127.0.0.1:54692_solr) [n:127.0.0.1:54692_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4381411 INFO  (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4381417 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 4381418 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65526_solr
   [junit4]   2> 4381418 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 4381419 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:54692_solr]
   [junit4]   2> 4381419 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 4381419 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 4381420 INFO  (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4381423 INFO  (zkCallback-7749-thread-1-processing-n:127.0.0.1:54692_solr) [n:127.0.0.1:54692_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4381424 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4381425 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 4381426 DEBUG (ScheduledTrigger-12632-thread-1) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 4381508 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65526.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381515 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65526.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381515 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65526.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381516 INFO  (jetty-launcher-7733-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A4833FA978AF33C2-001/tempDir-001/node2/.
   [junit4]   2> 4381527 INFO  (jetty-launcher-7733-thread-1) [n:127.0.0.1:54692_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54692.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381533 INFO  (jetty-launcher-7733-thread-1) [n:127.0.0.1:54692_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54692.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381533 INFO  (jetty-launcher-7733-thread-1) [n:127.0.0.1:54692_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54692.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381534 INFO  (jetty-launcher-7733-thread-1) [n:127.0.0.1:54692_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A4833FA978AF33C2-001/tempDir-001/node1/.
   [junit4]   2> 4381551 INFO  (zkConnectionManagerCallback-7758-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381555 INFO  (zkConnectionManagerCallback-7762-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381557 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4381558 INFO  (SUITE-TriggerIntegrationTest-seed#[A4833FA978AF33C2]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59561/solr ready
   [junit4]   2> 4381673 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testListeners
   [junit4]   2> 4381674 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 4381675 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 4381675 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 4381679 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4381680 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4381680 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 4381698 INFO  (qtp373898136-34667) [n:127.0.0.1:54692_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 4381700 INFO  (qtp1706134196-34675) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4381701 DEBUG (qtp1706134196-34663) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4381702 INFO  (qtp1706134196-34663) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=16
   [junit4]   2> 4381702 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 4381706 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:54692_solr, 127.0.0.1:65526_solr]
   [junit4]   2> 4381706 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction}], enabled=true}
   [junit4]   2> 4381706 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 4381709 INFO  (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 4381710 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4381711 DEBUG (ScheduledTrigger-12632-thread-1) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4381711 DEBUG (ScheduledTrigger-12632-thread-1) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 4381711 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 4381713 INFO  (qtp373898136-34669) [n:127.0.0.1:54692_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4381714 INFO  (qtp1706134196-34675) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4381714 DEBUG (qtp1706134196-34665) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4381715 INFO  (qtp1706134196-34665) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 4381716 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 4381717 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:54692_solr, 127.0.0.1:65526_solr]
   [junit4]   2> 4381717 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction}], enabled=true}
   [junit4]   2> 4381717 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 4381717 INFO  (qtp373898136-34671) [n:127.0.0.1:54692_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4381718 INFO  (qtp1706134196-34665) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4381718 DEBUG (qtp1706134196-34676) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4381719 INFO  (qtp1706134196-34676) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 4381720 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4381720 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 4381720 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4381721 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 4381722 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4381722 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4381722 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:54692_solr, 127.0.0.1:65526_solr]
   [junit4]   2> 4381722 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 4381722 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction}], enabled=true}
   [junit4]   2> 4381722 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 4381722 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4bbdd0a9{/solr,null,AVAILABLE}
   [junit4]   2> 4381723 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1abaad4c{HTTP/1.1,[http/1.1]}{127.0.0.1:49115}
   [junit4]   2> 4381723 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.Server Started @4389990ms
   [junit4]   2> 4381723 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49115}
   [junit4]   2> 4381723 ERROR (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4381723 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4381723 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 4381723 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4381724 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4381724 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-13T00:09:48.061Z
   [junit4]   2> 4381724 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 4381725 INFO  (zkConnectionManagerCallback-7764-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381726 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4381731 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10077082ba40009, likely client has closed socket
   [junit4]   2> 4381733 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59561/solr
   [junit4]   2> 4381735 INFO  (zkConnectionManagerCallback-7768-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381738 INFO  (zkConnectionManagerCallback-7770-thread-1-processing-n:127.0.0.1:49115_solr) [n:127.0.0.1:49115_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4381749 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4381752 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4381756 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49115_solr
   [junit4]   2> 4381757 INFO  (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4381757 INFO  (zkCallback-7749-thread-1-processing-n:127.0.0.1:54692_solr) [n:127.0.0.1:54692_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4381760 INFO  (zkCallback-7769-thread-1-processing-n:127.0.0.1:49115_solr) [n:127.0.0.1:49115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4381760 INFO  (zkCallback-7761-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4381852 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49115.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381869 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49115.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381869 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49115.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4381870 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:49115_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A4833FA978AF33C2-001/tempDir-001/node3/.
   [junit4]   2> 4382719 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4382720 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4382720 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:49115_solr at time 1997025749559703
   [junit4]   2> 4383726 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4383726 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4384728 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4384728 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 4384728 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL firing registered processor for nodes: [127.0.0.1:49115_solr] added at times [1997025749559703], now=1997027758778807
   [junit4]   2> 4384729 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"71848ca68e997T6wcqvo5rjgcxxbo2jk3jqoczk",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1997025749559703,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1997025749559703],
   [junit4]   2>     "nodeNames":["127.0.0.1:49115_solr"]}}
   [junit4]   2> 4384731 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerL]
   [junit4]   2> 4384732 INFO  (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 4384734 DEBUG (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"71848ca68e997T6wcqvo5rjgcxxbo2jk3jqoczk",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1997025749559703,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1997025749559703],
   [junit4]   2>     "_enqueue_time_":1997027761518102,
   [junit4]   2>     "nodeNames":["127.0.0.1:49115_solr"]}}
   [junit4]   2> 4384740 DEBUG (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerL after 5000ms
   [junit4]   2> 4384740 DEBUG (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 6 ms for event id=71848ca68e997T6wcqvo5rjgcxxbo2jk3jqoczk
   [junit4]   2> 4386743 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T15:27:37-06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 4386745 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4386745 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4386745 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 4386745 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@40015341{/solr,null,AVAILABLE}
   [junit4]   2> 4386746 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2ac27b57{HTTP/1.1,[http/1.1]}{127.0.0.1:58691}
   [junit4]   2> 4386746 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.e.j.s.Server Started @4395014ms
   [junit4]   2> 4386746 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58691}
   [junit4]   2> 4386746 ERROR (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4386747 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 4386747 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4386747 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4386747 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-02-13T00:09:53.084Z
   [junit4]   2> 4386749 INFO  (zkConnectionManagerCallback-7774-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4386750 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4386759 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59561/solr
   [junit4]   2> 4386761 INFO  (zkConnectionManagerCallback-7778-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4386765 INFO  (zkConnectionManagerCallback-7780-thread-1-processing-n:127.0.0.1:58691_solr) [n:127.0.0.1:58691_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4386787 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4386791 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4386795 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58691_solr
   [junit4]   2> 4386798 INFO  (zkCallback-7769-thread-1-processing-n:127.0.0.1:49115_solr) [n:127.0.0.1:49115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4386798 INFO  (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4386798 INFO  (zkCallback-7761-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4386798 INFO  (zkCallback-7749-thread-1-processing-n:127.0.0.1:54692_solr) [n:127.0.0.1:54692_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4386801 INFO  (zkCallback-7779-thread-1-processing-n:127.0.0.1:58691_solr) [n:127.0.0.1:58691_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4386889 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58691.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4386903 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58691.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4386903 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58691.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ac8eee4
   [junit4]   2> 4386905 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [n:127.0.0.1:58691_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_A4833FA978AF33C2-001/tempDir-001/node4/.
   [junit4]   2> 4389740 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4389741 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 4389741 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:58691_solr at time 1997032773047026
   [junit4]   2> 4390743 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4390743 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 4391744 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerL
   [junit4]   2> 4391744 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 4391744 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerL firing registered processor for nodes: [127.0.0.1:58691_solr] added at times [1997032773047026], now=1997034777322033
   [junit4]   2> 4391744 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"7184a6d0ad2f2T6wcqvo5rjgcxxbo2jk3jqoczm",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1997032773047026,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1997032773047026],
   [junit4]   2>     "nodeNames":["127.0.0.1:58691_solr"]}}
   [junit4]   2> 4391745 DEBUG (ScheduledTrigger-12632-thread-2) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerL]
   [junit4]   2> 4391745 DEBUG (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"7184a6d0ad2f2T6wcqvo5rjgcxxbo2jk3jqoczm",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1997032773047026,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1997032773047026],
   [junit4]   2>     "_enqueue_time_":1997034777588419,
   [junit4]   2>     "nodeNames":["127.0.0.1:58691_solr"]}}
   [junit4]   2> 4391746 WARN  (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: test1 for trigger event: {
   [junit4]   2>   "id":"7184a6d0ad2f2T6wcqvo5rjgcxxbo2jk3jqoczm",
   [junit4]   2>   "source":"node_added_triggerL",
   [junit4]   2>   "eventTime":1997032773047026,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[1997032773047026],
   [junit4]   2>     "_enqueue_time_":1997034777588419,
   [junit4]   2>     "nodeNames":["127.0.0.1:58691_solr"]}}
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:302)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.RuntimeException: failure
   [junit4]   2> at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestDummyAction.process(TriggerIntegrationTest.java:989)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:299)
   [junit4]   2> ... 6 more
   [junit4]   2> 4391746 DEBUG (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerL after 5000ms
   [junit4]   2> 4391747 DEBUG (AutoscalingActionExecutor-12633-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 1 ms for event id=7184a6d0ad2f2T6wcqvo5rjgcxxbo2jk3jqoczm
   [junit4]   2> 4393746 INFO  (TEST-TriggerIntegrationTest.testListeners-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testListeners
   [junit4]   2> 4394019 INFO  (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[A4833FA978AF33C2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testContinueTriggersOnOverseerRestart
   [junit4]   2> 4394020 INFO  (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[A4833FA978AF33C2]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 6
   [junit4]   2> 4394020 DEBUG (zkCallback-7751-thread-1-processing-n:127.0.0.1:65526_solr) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 4394021 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 4394023 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 4394023 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 4394024 DEBUG (OverseerAutoScalingTriggerThread-72188471013539846-127.0.0.1:65526_solr-n_0000000000) [n:127.0.0.1:65526_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 4394029 INFO  (qtp30582853-34727) [n:127.0.0.1:49115_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4394042 INFO  (qtp30582853-34727) [n:127.0.0.1:49115_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=17
   [junit4]   2> 4394047 INFO  (qtp30582853-34729) [n:127.0.0.1:49115_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4394049 INFO  (qtp373898136-34671) [n:127.0.0.1:54692_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4394059 INFO  (qtp1245065177-34753) [n:127.0.0.1:58691_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=8
   [junit4]   2> 4394060 INFO  (qtp1706134196-34676) [n:127.0.0.1:65526_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4394061 DEBUG (qtp30582853-34731) [n:127.0.0.1:49115_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 4394062 INFO  (qtp30582853-34731) [n:127.0.0.1:49115_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=17
   [junit4]   2> 4394062 DEBUG (z