[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 40 - Unstable!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 40 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/40/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelCommitStream

Error Message:
expected:<5> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<2>
        at __randomizedtesting.SeedInfo.seed([9D5BB385135C4C0C:BDB1D1858F1DA140]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelCommitStream(StreamExpressionTest.java:4582)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13791 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.StreamExpressionTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/init-core-data-001
   [junit4]   2> 34034 WARN  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 34034 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 34035 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 34037 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001
   [junit4]   2> 34043 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 34044 INFO  (Thread-42) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 34044 INFO  (Thread-42) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 34132 ERROR (Thread-42) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 34144 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:48780
   [junit4]   2> 34241 INFO  (jetty-launcher-10-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 34241 INFO  (jetty-launcher-10-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 34242 INFO  (jetty-launcher-10-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 34243 INFO  (jetty-launcher-10-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@697c3cb1{/solr,null,AVAILABLE}
   [junit4]   2> 34244 INFO  (jetty-launcher-10-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@766a7c06{/solr,null,AVAILABLE}
   [junit4]   2> 34246 INFO  (jetty-launcher-10-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7985ea22{HTTP/1.1,[http/1.1]}{127.0.0.1:44488}
   [junit4]   2> 34246 INFO  (jetty-launcher-10-thread-2) [    ] o.e.j.s.Server Started @36220ms
   [junit4]   2> 34246 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44488}
   [junit4]   2> 34247 ERROR (jetty-launcher-10-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 34247 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 34247 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 34247 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 34247 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-15T03:08:26.672Z
   [junit4]   2> 34247 INFO  (jetty-launcher-10-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11a33dc2{HTTP/1.1,[http/1.1]}{127.0.0.1:64680}
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.e.j.s.Server Started @36222ms
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64680}
   [junit4]   2> 34248 ERROR (jetty-launcher-10-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 34248 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-15T03:08:26.673Z
   [junit4]   2> 34251 INFO  (jetty-launcher-10-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7a9b98f3{/solr,null,AVAILABLE}
   [junit4]   2> 34253 INFO  (jetty-launcher-10-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55adcc7c{/solr,null,AVAILABLE}
   [junit4]   2> 34254 INFO  (jetty-launcher-10-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1947656a{HTTP/1.1,[http/1.1]}{127.0.0.1:52469}
   [junit4]   2> 34254 INFO  (jetty-launcher-10-thread-4) [    ] o.e.j.s.Server Started @36228ms
   [junit4]   2> 34254 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52469}
   [junit4]   2> 34254 ERROR (jetty-launcher-10-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 34254 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 34254 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 34255 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 34255 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-15T03:08:26.680Z
   [junit4]   2> 34258 INFO  (jetty-launcher-10-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@484ba48a{HTTP/1.1,[http/1.1]}{127.0.0.1:50144}
   [junit4]   2> 34258 INFO  (jetty-launcher-10-thread-3) [    ] o.e.j.s.Server Started @36232ms
   [junit4]   2> 34258 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50144}
   [junit4]   2> 34259 ERROR (jetty-launcher-10-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 34259 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 34259 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 34259 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 34259 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-15T03:08:26.684Z
   [junit4]   2> 34281 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 34282 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 34283 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 34283 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 34304 INFO  (jetty-launcher-10-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48780/solr
   [junit4]   2> 34310 INFO  (jetty-launcher-10-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48780/solr
   [junit4]   2> 34314 INFO  (jetty-launcher-10-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48780/solr
   [junit4]   2> 34315 INFO  (jetty-launcher-10-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48780/solr
   [junit4]   2> 34331 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d4437fe0b0005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 34414 WARN  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 34415 WARN  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 34417 WARN  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 34420 WARN  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 34448 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 34449 INFO  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 34451 INFO  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 34452 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52469_solr
   [junit4]   2> 34454 INFO  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 34455 INFO  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64680_solr
   [junit4]   2> 34460 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 34461 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.c.Overseer Overseer (id=98309774148042762-127.0.0.1:52469_solr-n_0000000000) starting
   [junit4]   2> 34468 INFO  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50144_solr
   [junit4]   2> 34468 INFO  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44488_solr
   [junit4]   2> 34463 INFO  (zkCallback-32-thread-1-processing-n:127.0.0.1:64680_solr) [n:127.0.0.1:64680_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 34469 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 34469 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:52469_solr) [n:127.0.0.1:52469_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 34476 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 34478 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:52469_solr) [n:127.0.0.1:52469_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 34478 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 34526 INFO  (zkCallback-32-thread-1-processing-n:127.0.0.1:64680_solr) [n:127.0.0.1:64680_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 34551 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52469_solr
   [junit4]   2> 34560 INFO  (zkCallback-32-thread-1-processing-n:127.0.0.1:64680_solr) [n:127.0.0.1:64680_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 34570 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:52469_solr) [n:127.0.0.1:52469_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 34571 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 34596 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 34700 INFO  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50144.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34713 INFO  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64680.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34721 INFO  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64680.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34721 INFO  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_64680.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34738 INFO  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44488.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34744 INFO  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44488.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34744 INFO  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44488.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34758 INFO  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50144.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34758 INFO  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50144.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34759 INFO  (jetty-launcher-10-thread-1) [n:127.0.0.1:64680_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node1/.
   [junit4]   2> 34769 INFO  (jetty-launcher-10-thread-2) [n:127.0.0.1:44488_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node2/.
   [junit4]   2> 34771 INFO  (jetty-launcher-10-thread-3) [n:127.0.0.1:50144_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node4/.
   [junit4]   2> 34806 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52469.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34833 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52469.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34835 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52469.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 34837 INFO  (jetty-launcher-10-thread-4) [n:127.0.0.1:52469_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node3/.
   [junit4]   2> 34907 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d4437fe0b000d, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 34914 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 34916 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 34942 INFO  (qtp1845760834-176) [n:127.0.0.1:52469_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 34957 INFO  (OverseerThreadFactory-92-thread-1-processing-n:127.0.0.1:52469_solr) [n:127.0.0.1:52469_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 35074 INFO  (OverseerStateUpdate-98309774148042762-127.0.0.1:52469_solr-n_0000000000) [n:127.0.0.1:52469_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:44488/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 35079 INFO  (OverseerStateUpdate-98309774148042762-127.0.0.1:52469_solr-n_0000000000) [n:127.0.0.1:52469_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"collection1_shard2_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:50144/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 35285 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 35285 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 35286 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 35286 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 35451 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 35452 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 35452 INFO  (zkCallback-30-thread-2-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 35452 INFO  (zkCallback-31-thread-2-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36323 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 36335 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
   [junit4]   2> 36336 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 36350 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 36431 WARN  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.GeoHashField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 36436 WARN  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.LatLonType]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 36436 WARN  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.LatLonType]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 36440 WARN  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 36440 WARN  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 36440 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 36442 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 36443 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44488.solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 36443 WARN  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 36443 WARN  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 36443 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 36443 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 36444 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50144.solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f4c25c8
   [junit4]   2> 36444 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 36444 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 36444 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node4/collection1_shard2_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node4/./collection1_shard2_replica_n1/data/]
   [junit4]   2> 36444 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node2/collection1_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001/tempDir-001/node2/./collection1_shard1_replica_n1/data/]
   [junit4]   2> 36507 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 36508 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 36509 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 36509 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 36510 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 36510 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 36510 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4a07727f[collection1_shard2_replica_n1] main]
   [junit4]   2> 36511 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 36511 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 36514 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@8c13c3f[collection1_shard1_replica_n1] main]
   [junit4]   2> 36516 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 36517 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 36517 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 36517 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 36518 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 36518 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 36521 INFO  (searcherExecutor-101-thread-1-processing-n:127.0.0.1:44488_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@8c13c3f[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 36521 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572956388926160896
   [junit4]   2> 36521 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572956388926160896
   [junit4]   2> 36525 INFO  (searcherExecutor-102-thread-1-processing-n:127.0.0.1:50144_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@4a07727f[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 36579 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 36579 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 36579 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/
   [junit4]   2> 36582 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 36582 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 36582 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/
   [junit4]   2> 36583 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 36583 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 36583 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 36583 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/ has no replicas
   [junit4]   2> 36583 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 36583 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 36606 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/ shard1
   [junit4]   2> 36611 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 36716 INFO  (zkCallback-31-thread-2-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36716 INFO  (zkCallback-30-thread-2-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36716 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36716 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36761 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 36768 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 36775 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1491
   [junit4]   2> 36779 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1493
   [junit4]   2> 36798 INFO  (qtp1845760834-176) [n:127.0.0.1:52469_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 36882 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36882 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36882 INFO  (zkCallback-31-thread-2-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36882 INFO  (zkCallback-30-thread-2-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 36959 INFO  (OverseerCollectionConfigSetProcessor-98309774148042762-127.0.0.1:52469_solr-n_0000000000) [n:127.0.0.1:52469_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 37801 INFO  (qtp1845760834-176) [n:127.0.0.1:52469_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2859
   [junit4]   2> 37806 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):90
   [junit4]   2> 37808 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
   [junit4]   2> 37833 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFacetStream
   [junit4]   2> 37871 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1572956390336495616)} 0 11
   [junit4]   2> 37875 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390345932800,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 37875 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 37876 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390346981376,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 37876 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 37883 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 37883 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 8
   [junit4]   2> 37892 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 37892 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 16
   [junit4]   2> 37893 INFO  (qtp2027612163-167) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1572956390319718400),commit=} 0 55
   [junit4]   2> 37919 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[0 (1572956390377390080), 4 (1572956390392070144), 1 (1572956390392070145), 8 (1572956390392070146)]} 0 14
   [junit4]   2> 37933 INFO  (qtp1313660971-191) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[2 (1572956390383681536), 3 (1572956390405701632), 5 (1572956390406750208), 6 (1572956390406750209), 7 (1572956390406750210), 9 (1572956390406750211)]} 0 22
   [junit4]   2> 37942 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390416187392,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 37947 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390421430272,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 37948 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2981bcaa commitCommandVersion:1572956390416187392
   [junit4]   2> 37949 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2a359849 commitCommandVersion:1572956390421430272
   [junit4]   2> 38025 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d4e2110[collection1_shard1_replica_n1] main]
   [junit4]   2> 38026 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38027 INFO  (searcherExecutor-101-thread-1-processing-n:127.0.0.1:44488_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@2d4e2110[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c4)))}
   [junit4]   2> 38027 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 80
   [junit4]   2> 38029 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@160dda73[collection1_shard2_replica_n1] main]
   [junit4]   2> 38031 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38032 INFO  (searcherExecutor-102-thread-1-processing-n:127.0.0.1:50144_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@160dda73[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c6)))}
   [junit4]   2> 38032 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 90
   [junit4]   2> 38068 INFO  (qtp1313660971-193) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=} 0 131
   [junit4]   2> 38083 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 38084 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 38159 INFO  (qtp2027612163-180) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"facet_0":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110511&isShard=true&wt=javabin} hits=4 status=0 QTime=19
   [junit4]   2> 38159 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"facet_0":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110511&isShard=true&wt=javabin} hits=6 status=0 QTime=28
   [junit4]   2> 38167 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"facet_0":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&_stateVer_=collection1:4&fl=a_s,a_i,a_f&bucketSizeLimit=100&sort=a_s+asc&rows=0&wt=javabin&version=2} hits=10 status=0 QTime=80
   [junit4]   2> 38177 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 38178 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 38190 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"facet_0":"desc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110609&isShard=true&wt=javabin} hits=4 status=0 QTime=2
   [junit4]   2> 38191 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"facet_0":"desc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110609&isShard=true&wt=javabin} hits=6 status=0 QTime=4
   [junit4]   2> 38192 INFO  (qtp2027612163-190) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"facet_0":"desc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&_stateVer_=collection1:4&fl=a_s,a_i,a_f&bucketSizeLimit=100&sort=a_s+asc&rows=0&wt=javabin&version=2} hits=10 status=0 QTime=8
   [junit4]   2> 38199 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 38200 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 38204 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"desc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110626&isShard=true&wt=javabin} hits=4 status=0 QTime=1
   [junit4]   2> 38205 INFO  (qtp1313660971-193) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"desc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110626&isShard=true&wt=javabin} hits=6 status=0 QTime=2
   [junit4]   2> 38207 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"desc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&_stateVer_=collection1:4&fl=a_s,a_i,a_f&bucketSizeLimit=100&sort=a_s+asc&rows=0&wt=javabin&version=2} hits=10 status=0 QTime=6
   [junit4]   2> 38219 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 38221 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 38225 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110647&isShard=true&wt=javabin} hits=4 status=0 QTime=1
   [junit4]   2> 38228 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110647&isShard=true&wt=javabin} hits=6 status=0 QTime=1
   [junit4]   2> 38229 INFO  (qtp2027612163-165) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&_stateVer_=collection1:4&fl=a_s,a_i,a_f&bucketSizeLimit=100&sort=a_s+asc&rows=0&wt=javabin&version=2} hits=10 status=0 QTime=6
   [junit4]   2> 38236 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 38240 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 38248 INFO  (qtp1313660971-185) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&rows=0&version=2&q=blahhh&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110667&isShard=true&wt=javabin} hits=0 status=0 QTime=1
   [junit4]   2> 38248 INFO  (qtp2027612163-167) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1:4&_facet_={}&fl=id&shards.purpose=1048580&start=0&fsv=true&sort=a_s+asc&shard.url=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&rows=0&version=2&q=blahhh&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&bucketSizeLimit=100&NOW=1500088110667&isShard=true&wt=javabin} hits=0 status=0 QTime=0
   [junit4]   2> 38249 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q=blahhh&json.facet={"a_s":{"type":"terms","field":"a_s","limit":100,"sort":{"index":"asc"},"facet":{"facet_0":"sum(a_i)","facet_1":"sum(a_f)","facet_2":"min(a_i)","facet_3":"min(a_f)","facet_4":"max(a_i)","facet_5":"max(a_f)","facet_6":"avg(a_i)","facet_7":"avg(a_f)"}}}&_stateVer_=collection1:4&fl=a_s,a_i,a_f&bucketSizeLimit=100&sort=a_s+asc&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=6
   [junit4]   2> 38252 INFO  (TEST-StreamExpressionTest.testFacetStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testFacetStream
   [junit4]   2> 38279 INFO  (TEST-StreamExpressionTest.testFetchStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFetchStream
   [junit4]   2> 38283 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@44252b74[collection1_shard2_replica_n1] realtime]
   [junit4]   2> 38284 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c2b20dd[collection1_shard1_replica_n1] realtime]
   [junit4]   2> 38284 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1572956390772703232)} 0 2
   [junit4]   2> 38286 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390776897536,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 38286 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2a359849 commitCommandVersion:1572956390776897536
   [junit4]   2> 38286 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2764de72[collection1_shard1_replica_n1] main]
   [junit4]   2> 38287 INFO  (searcherExecutor-101-thread-1-processing-n:127.0.0.1:44488_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@2764de72[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 38293 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390784237568,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 38294 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2981bcaa commitCommandVersion:1572956390784237568
   [junit4]   2> 38294 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38294 INFO  (qtp2027612163-169) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 8
   [junit4]   2> 38295 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1c56ecfd[collection1_shard2_replica_n1] main]
   [junit4]   2> 38296 INFO  (searcherExecutor-102-thread-1-processing-n:127.0.0.1:50144_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@1c56ecfd[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 38297 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38297 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:50144/solr/collection1_shard2_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
   [junit4]   2> 38298 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1572956390770606080),commit=} 0 17
   [junit4]   2> 38302 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[2 (1572956390790529024), 3 (1572956390791577600), 5 (1572956390792626176), 6 (1572956390792626177), 7 (1572956390792626178), 9 (1572956390792626179)]} 0 3
   [junit4]   2> 38305 INFO  (qtp2027612163-186) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[0 (1572956390793674752), 4 (1572956390795771904), 1 (1572956390796820480), 8 (1572956390796820481)]} 0 3
   [junit4]   2> 38308 INFO  (qtp1313660971-185) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390799966208,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 38308 INFO  (qtp1313660971-185) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2981bcaa commitCommandVersion:1572956390799966208
   [junit4]   2> 38309 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390801014784,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 38309 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2a359849 commitCommandVersion:1572956390801014784
   [junit4]   2> 38322 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@669cd788[collection1_shard1_replica_n1] main]
   [junit4]   2> 38324 INFO  (searcherExecutor-101-thread-1-processing-n:127.0.0.1:44488_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@669cd788[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):c4)))}
   [junit4]   2> 38325 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38325 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 16
   [junit4]   2> 38326 INFO  (qtp1313660971-185) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@616826d2[collection1_shard2_replica_n1] main]
   [junit4]   2> 38328 INFO  (searcherExecutor-102-thread-1-processing-n:127.0.0.1:50144_solr x:collection1_shard2_replica_n1 s:shard2 c:collection1 r:core_node2) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@616826d2[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):c6)))}
   [junit4]   2> 38329 INFO  (qtp1313660971-185) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38329 INFO  (qtp1313660971-185) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 21
   [junit4]   2> 38331 INFO  (qtp2027612163-180) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=} 0 23
   [junit4]   2> 38338 INFO  (TEST-StreamExpressionTest.testFetchStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 38340 INFO  (TEST-StreamExpressionTest.testFetchStream-seed#[9D5BB385135C4C0C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48780/solr ready
   [junit4]   2> 38346 INFO  (qtp2027612163-190) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&wt=json&version=2.2} hits=4 status=0 QTime=0
   [junit4]   2> 38347 INFO  (qtp1313660971-191) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&wt=json&version=2.2} hits=6 status=0 QTime=0
   [junit4]   2> 38359 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+0+2&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=2
   [junit4]   2> 38365 INFO  (qtp1313660971-187) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+0+2&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38371 INFO  (qtp2027612163-166) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+3+4&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38371 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+3+4&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38376 INFO  (qtp1313660971-193) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+1+5&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38377 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+1+5&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38381 INFO  (qtp2027612163-190) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+6+7&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 38381 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+6+7&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=2 status=0 QTime=0
   [junit4]   2> 38384 INFO  (qtp1313660971-191) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+8+9&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38388 INFO  (qtp2027612163-186) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+8+9&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=2&wt=json&version=2.2} hits=1 status=0 QTime=4
   [junit4]   2> 38393 INFO  (qtp2027612163-164) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&wt=json&version=2.2} hits=4 status=0 QTime=0
   [junit4]   2> 38393 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&wt=json&version=2.2} hits=6 status=0 QTime=0
   [junit4]   2> 38397 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+0+2+3&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38399 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+0+2+3&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=2 status=0 QTime=0
   [junit4]   2> 38405 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+4+1+5&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38408 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+4+1+5&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=2 status=0 QTime=0
   [junit4]   2> 38413 INFO  (qtp2027612163-168) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+6+7+8&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38415 INFO  (qtp1313660971-192) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+6+7+8&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=2 status=0 QTime=0
   [junit4]   2> 38419 INFO  (qtp1313660971-188) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.c.S.Request [collection1_shard2_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+9&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 38421 INFO  (qtp2027612163-190) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1]  webapp=/solr path=/select params={q={!+df%3Da_i+q.op%3DOR+cache%3Dfalse+}+9&distrib=false&fl=subject,a_i,_version_&sort=_version_+desc&rows=3&wt=json&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 38425 INFO  (qtp2027612163-190) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[99 (1572956390921601024)]} 0 1
   [junit4]   2> 38427 INFO  (qtp2027612163-167) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390924746752,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 38427 INFO  (qtp2027612163-167) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2a359849 commitCommandVersion:1572956390924746752
   [junit4]   2> 38430 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572956390927892480,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 38430 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 38435 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 38436 INFO  (qtp1313660971-189) [n:127.0.0.1:50144_solr c:collection1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44488/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 6
   [junit4]   2> 38445 INFO  (qtp2027612163-167) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher

[...truncated too long message...]

1 s:shard2 r:core_node2 x:collection1_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=944791282
   [junit4]   2> 143599 INFO  (jetty-closer-11-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@697c3cb1{/solr,null,UNAVAILABLE}
   [junit4]   2> 143607 INFO  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 143608 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 143609 INFO  (jetty-closer-11-thread-4) [    ] o.a.s.c.Overseer Overseer (id=98309774148042762-127.0.0.1:52469_solr-n_0000000000) closing
   [junit4]   2> 143610 INFO  (OverseerStateUpdate-98309774148042762-127.0.0.1:52469_solr-n_0000000000) [n:127.0.0.1:52469_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:52469_solr
   [junit4]   2> 143613 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d4437fe0b000a, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 143615 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (1)
   [junit4]   2> 143616 INFO  (coreCloseExecutor-293-thread-1) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_n1, tag=1920870892
   [junit4]   2> 143618 INFO  (coreCloseExecutor-293-thread-1) [n:127.0.0.1:44488_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1920870892
   [junit4]   2> 143615 INFO  (zkCallback-31-thread-2-processing-n:127.0.0.1:44488_solr) [n:127.0.0.1:44488_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44488_solr
   [junit4]   2> 143625 WARN  (zkCallback-29-thread-1-processing-n:127.0.0.1:52469_solr) [n:127.0.0.1:52469_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 143626 INFO  (jetty-closer-11-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7a9b98f3{/solr,null,UNAVAILABLE}
   [junit4]   2> 143626 WARN  (zkCallback-30-thread-1-processing-n:127.0.0.1:50144_solr) [n:127.0.0.1:50144_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 143626 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d4437fe0b000c, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 143626 INFO  (jetty-closer-11-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@55adcc7c{/solr,null,UNAVAILABLE}
   [junit4]   2> 145126 INFO  (jetty-closer-11-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@766a7c06{/solr,null,UNAVAILABLE}
   [junit4]   2> 145129 ERROR (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 145131 INFO  (SUITE-StreamExpressionTest-seed#[9D5BB385135C4C0C]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48780 48780
   [junit4]   2> 145959 INFO  (Thread-42) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48780 48780
   [junit4]   2> 145960 WARN  (Thread-42) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/aliases.json
   [junit4]   2> 91 /solr/clusterprops.json
   [junit4]   2> 12 /solr/configs/conf
   [junit4]   2> 7 /solr/configs/ml
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/clusterstate.json
   [junit4]   2> 40 /solr/collections/destinationCollection/state.json
   [junit4]   2> 26 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 20 /solr/collections/destination/state.json
   [junit4]   2> 20 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 20 /solr/collections/mainCorpus/state.json
   [junit4]   2> 18 /solr/collections/workQueue/state.json
   [junit4]   2> 10 /solr/collections/collection1/state.json
   [junit4]   2> 10 /solr/collections/uknownCollection/state.json
   [junit4]   2> 10 /solr/collections/modelCollection/state.json
   [junit4]   2> 10 /solr/collections/checkpointCollection/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98309774148042762-127.0.0.1:52469_solr-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/live_nodes
   [junit4]   2> 91 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_9D5BB385135C4C0C-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=BlockTreeOrds(blocksize=128), expr_s=PostingsFormat(name=Asserting), terms_ss=FST50, join2_s=FST50, field_s=PostingsFormat(name=Memory), multiDefault=FST50, subject=FST50, a1_s=PostingsFormat(name=Memory), a2_s=PostingsFormat(name=Asserting), level1_s=PostingsFormat(name=Asserting), body_t=BlockTreeOrds(blocksize=128), s_multi=PostingsFormat(name=Asserting), col_s=PostingsFormat(name=Memory), level2_s=PostingsFormat(name=Memory), a_ss=FST50, a_s=PostingsFormat(name=Asserting), tv_text=PostingsFormat(name=Memory), term_s=FST50, a_t=FST50, text_s=PostingsFormat(name=Asserting), id=BlockTreeOrds(blocksize=128), text=PostingsFormat(name=Asserting), featureSet_s=PostingsFormat(name=Memory), checkpoint_ss=BlockTreeOrds(blocksize=128), test_t=BlockTreeOrds(blocksize=128), ident_s=PostingsFormat(name=Memory), whitetok=BlockTreeOrds(blocksize=128), side_s=BlockTreeOrds(blocksize=128)}, docValues:{name_s=DocValuesFormat(name=Memory), expr_s=DocValuesFormat(name=Asserting), iteration_i=DocValuesFormat(name=Memory), terms_ss=DocValuesFormat(name=Direct), join2_s=DocValuesFormat(name=Direct), multiDefault=DocValuesFormat(name=Direct), a_f=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Asserting), falseNegative_i=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Memory), level1_s=DocValuesFormat(name=Asserting), a_s=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Memory), alpha_d=DocValuesFormat(name=Memory), i_multi=DocValuesFormat(name=Memory), idf_d=DocValuesFormat(name=Lucene70), b_ls=DocValuesFormat(name=Lucene70), checkpoint_ss=DocValuesFormat(name=Memory), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Direct), error_d=DocValuesFormat(name=Asserting), side_s=DocValuesFormat(name=Memory), truePositive_i=DocValuesFormat(name=Asserting), miles_i=DocValuesFormat(name=Lucene70), field_s=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Direct), join1_i=DocValuesFormat(name=Asserting), join3_i=DocValuesFormat(name=Memory), test_i=DocValuesFormat(name=Direct), falsePositive_i=DocValuesFormat(name=Lucene70), field_i=DocValuesFormat(name=Direct), s_multi=DocValuesFormat(name=Asserting), col_s=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Direct), score_f=DocValuesFormat(name=Asserting), term_s=DocValuesFormat(name=Direct), text_s=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Direct), idfs_ds=DocValuesFormat(name=Direct), weights_ds=DocValuesFormat(name=Memory), featureSet_s=DocValuesFormat(name=Lucene70), trueNegative_i=DocValuesFormat(name=Asserting), index_i=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Asserting), out_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1945, maxMBSortInHeap=5.21411583750359, sim=RandomSimilarity(queryNorm=true): {}, locale=de-CH, timezone=America/Tegucigalpa
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=76728688,total=211812352
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrJErrorHandling, SineEvaluatorTest, CloudSolrClientCacheTest, CommonParamsTest, CoalesceEvaluatorTest, ArrayEvaluatorTest, ArcSineEvaluatorTest, SolrExampleEmbeddedTest, TestClusteringResponse, CollectionAdminRequestRequiredParamsTest, SolrDocumentTest, TestJavaBinCodec, TestEmbeddedSolrServer, FieldAnalysisResponseTest, StreamExpressionTest]
   [junit4] Completed [59/135 (1!)] on J0 in 111.96s, 91 tests, 1 failure <<< FAILURES!

[...truncated 42744 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 41 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/41/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)  at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)  at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)  at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)  at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)  at sun.reflect.GeneratedConstructorAccessor149.newInstance(Unknown Source)  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)  at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)  at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
        at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)
        at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
        at sun.reflect.GeneratedConstructorAccessor149.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)
        at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)
        at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([4B5C98696D8196D8]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:296)
        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12681 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_4B5C98696D8196D8-001/init-core-data-001
   [junit4]   2> 2817007 WARN  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 2817007 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2817009 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2817010 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_4B5C98696D8196D8-001/tempDir-001
   [junit4]   2> 2817010 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2817011 INFO  (Thread-10194) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2817011 INFO  (Thread-10194) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2817013 ERROR (Thread-10194) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2817114 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:46921
   [junit4]   2> 2817127 INFO  (jetty-launcher-4767-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2817127 INFO  (jetty-launcher-4767-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2817131 INFO  (jetty-launcher-4767-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60e1648f{/solr,null,AVAILABLE}
   [junit4]   2> 2817131 INFO  (jetty-launcher-4767-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@272ba9c1{/solr,null,AVAILABLE}
   [junit4]   2> 2817132 INFO  (jetty-launcher-4767-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@d17915f{HTTP/1.1,[http/1.1]}{127.0.0.1:48835}
   [junit4]   2> 2817132 INFO  (jetty-launcher-4767-thread-1) [    ] o.e.j.s.Server Started @2819579ms
   [junit4]   2> 2817132 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48835}
   [junit4]   2> 2817132 INFO  (jetty-launcher-4767-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a0157c2{HTTP/1.1,[http/1.1]}{127.0.0.1:35947}
   [junit4]   2> 2817134 INFO  (jetty-launcher-4767-thread-2) [    ] o.e.j.s.Server Started @2819581ms
   [junit4]   2> 2817134 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35947}
   [junit4]   2> 2817134 ERROR (jetty-launcher-4767-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2817135 ERROR (jetty-launcher-4767-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2817135 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-15T11:15:17.153Z
   [junit4]   2> 2817136 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-15T11:15:17.153Z
   [junit4]   2> 2817142 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2817142 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2817173 INFO  (jetty-launcher-4767-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46921/solr
   [junit4]   2> 2817173 INFO  (jetty-launcher-4767-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46921/solr
   [junit4]   2> 2817310 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2817312 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:48835_solr
   [junit4]   2> 2817314 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.c.Overseer Overseer (id=98311688488812549-127.0.0.1:48835_solr-n_0000000000) starting
   [junit4]   2> 2817323 INFO  (jetty-launcher-4767-thread-2) [n:127.0.0.1:35947_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2817325 INFO  (jetty-launcher-4767-thread-2) [n:127.0.0.1:35947_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35947_solr
   [junit4]   2> 2817328 INFO  (zkCallback-4779-thread-1-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2817330 INFO  (zkCallback-4778-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2817333 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48835_solr
   [junit4]   2> 2817335 INFO  (zkCallback-4779-thread-1-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2817335 INFO  (zkCallback-4778-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2817443 INFO  (jetty-launcher-4767-thread-2) [n:127.0.0.1:35947_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35947.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2817445 INFO  (jetty-launcher-4767-thread-2) [n:127.0.0.1:35947_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35947.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2817445 INFO  (jetty-launcher-4767-thread-2) [n:127.0.0.1:35947_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35947.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2817446 INFO  (jetty-launcher-4767-thread-2) [n:127.0.0.1:35947_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_4B5C98696D8196D8-001/tempDir-001/node2/.
   [junit4]   2> 2817475 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48835.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2817477 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48835.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2817477 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48835.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2817478 INFO  (jetty-launcher-4767-thread-1) [n:127.0.0.1:48835_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_4B5C98696D8196D8-001/tempDir-001/node1/.
   [junit4]   2> 2817501 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2817503 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46921/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2817539 WARN  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2817548 WARN  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2817550 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2817564 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_serv1.vm3_59897_hdfs____2r6ycz/webapp
   [junit4]   2> 2818032 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@serv1-vm3:59897
   [junit4]   2> 2818142 WARN  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2818143 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2818152 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47502_datanode____.21doqt/webapp
   [junit4]   2> 2818604 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47502
   [junit4]   2> 2818682 WARN  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2818684 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2818705 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_64073_datanode____m2q451/webapp
   [junit4]   2> 2818728 INFO  (IPC Server handler 8 on 37218) [    ] BlockStateChange BLOCK* processReport: from storage DS-518fb1e0-e1f9-44d5-bb42-f72990f61ec6 node DatanodeRegistration(127.0.0.1:53435, datanodeUuid=d816404c-c6b6-44ba-9b14-7345a405666a, infoPort=56155, infoSecurePort=0, ipcPort=42163, storageInfo=lv=-56;cid=testClusterID;nsid=1725316182;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 2818728 INFO  (IPC Server handler 8 on 37218) [    ] BlockStateChange BLOCK* processReport: from storage DS-b4ff57ba-7135-48b4-b5cc-f1d5925fca7e node DatanodeRegistration(127.0.0.1:53435, datanodeUuid=d816404c-c6b6-44ba-9b14-7345a405666a, infoPort=56155, infoSecurePort=0, ipcPort=42163, storageInfo=lv=-56;cid=testClusterID;nsid=1725316182;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2819176 INFO  (SUITE-HdfsRecoveryZkTest-seed#[4B5C98696D8196D8]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:64073
   [junit4]   2> 2819287 INFO  (IPC Server handler 1 on 37218) [    ] BlockStateChange BLOCK* processReport: from storage DS-0daab4cb-b158-482b-bc80-ed1c628b77dd node DatanodeRegistration(127.0.0.1:62704, datanodeUuid=c99895af-0365-4d47-a2d0-ddc801955780, infoPort=49149, infoSecurePort=0, ipcPort=49107, storageInfo=lv=-56;cid=testClusterID;nsid=1725316182;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 2819287 INFO  (IPC Server handler 1 on 37218) [    ] BlockStateChange BLOCK* processReport: from storage DS-175e57dc-202f-490e-a625-6cc99ef8ffce node DatanodeRegistration(127.0.0.1:62704, datanodeUuid=c99895af-0365-4d47-a2d0-ddc801955780, infoPort=49149, infoSecurePort=0, ipcPort=49107, storageInfo=lv=-56;cid=testClusterID;nsid=1725316182;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2819381 INFO  (TEST-HdfsRecoveryZkTest.test-seed#[4B5C98696D8196D8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2819382 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2819385 INFO  (OverseerThreadFactory-13037-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.CreateCollectionCmd Create collection recoverytest
   [junit4]   2> 2819495 INFO  (OverseerStateUpdate-98311688488812549-127.0.0.1:48835_solr-n_0000000000) [n:127.0.0.1:48835_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"recoverytest",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"recoverytest_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:48835/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 2819499 INFO  (OverseerStateUpdate-98311688488812549-127.0.0.1:48835_solr-n_0000000000) [n:127.0.0.1:48835_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"recoverytest",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"recoverytest_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:35947/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 2819702 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2819702 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2819702 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2819703 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2819817 INFO  (zkCallback-4778-thread-2-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2819817 INFO  (zkCallback-4779-thread-1-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2819817 INFO  (zkCallback-4778-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2819818 INFO  (zkCallback-4779-thread-2-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2820731 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2820731 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2820738 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.s.IndexSchema [recoverytest_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2820738 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.s.IndexSchema [recoverytest_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2820741 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2820741 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'recoverytest_shard1_replica_n1' using configuration from collection recoverytest, trusted=true
   [junit4]   2> 2820741 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2820741 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'recoverytest_shard1_replica_n2' using configuration from collection recoverytest, trusted=true
   [junit4]   2> 2820741 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48835.solr.core.recoverytest.shard1.replica_n1' (registry 'solr.core.recoverytest.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2820742 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35947.solr.core.recoverytest.shard1.replica_n2' (registry 'solr.core.recoverytest.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@735d7e35
   [junit4]   2> 2820742 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://serv1-vm3:37218/data
   [junit4]   2> 2820742 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://serv1-vm3:37218/data
   [junit4]   2> 2820742 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2820742 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2820742 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2820742 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2820742 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.SolrCore [[recoverytest_shard1_replica_n2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_4B5C98696D8196D8-001/tempDir-001/node2/recoverytest_shard1_replica_n2], dataDir=[hdfs://serv1-vm3:37218/data/recoverytest/core_node2/data/]
   [junit4]   2> 2820742 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.SolrCore [[recoverytest_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_4B5C98696D8196D8-001/tempDir-001/node1/recoverytest_shard1_replica_n1], dataDir=[hdfs://serv1-vm3:37218/data/recoverytest/core_node1/data/]
   [junit4]   2> 2820743 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://serv1-vm3:37218/data/recoverytest/core_node2/data/snapshot_metadata
   [junit4]   2> 2820743 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://serv1-vm3:37218/data/recoverytest/core_node1/data/snapshot_metadata
   [junit4]   2> 2820749 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2820749 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2820749 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 2820750 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2820750 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2820758 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2820758 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2820759 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://serv1-vm3:37218/data/recoverytest/core_node2/data
   [junit4]   2> 2820759 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://serv1-vm3:37218/data/recoverytest/core_node1/data
   [junit4]   2> 2820778 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://serv1-vm3:37218/data/recoverytest/core_node2/data/index
   [junit4]   2> 2820778 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://serv1-vm3:37218/data/recoverytest/core_node1/data/index
   [junit4]   2> 2820783 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2820783 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2820783 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 2820783 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2820787 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2820787 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2820819 INFO  (IPC Server handler 9 on 37218) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53435 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b4ff57ba-7135-48b4-b5cc-f1d5925fca7e:NORMAL:127.0.0.1:53435|RBW], ReplicaUC[[DISK]DS-175e57dc-202f-490e-a625-6cc99ef8ffce:NORMAL:127.0.0.1:62704|RBW]]} size 72
   [junit4]   2> 2820819 INFO  (IPC Server handler 9 on 37218) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53435 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0daab4cb-b158-482b-bc80-ed1c628b77dd:NORMAL:127.0.0.1:62704|RBW], ReplicaUC[[DISK]DS-b4ff57ba-7135-48b4-b5cc-f1d5925fca7e:NORMAL:127.0.0.1:53435|RBW]]} size 72
   [junit4]   2> 2820823 INFO  (IPC Server handler 1 on 37218) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:62704 is added to blk_1073741825_1001 size 72
   [junit4]   2> 2820824 INFO  (IPC Server handler 4 on 37218) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:62704 is added to blk_1073741826_1002 size 72
   [junit4]   2> 2821313 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2821313 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2821313 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2821317 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2821317 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2821317 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2821322 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2821322 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2821328 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2821328 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2821340 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@57f8ed70[recoverytest_shard1_replica_n2] main]
   [junit4]   2> 2821340 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ad5da8[recoverytest_shard1_replica_n1] main]
   [junit4]   2> 2821342 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2821342 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2821342 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2821343 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2821343 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2821344 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2821344 INFO  (searcherExecutor-13043-thread-1-processing-n:127.0.0.1:35947_solr x:recoverytest_shard1_replica_n2 s:shard1 c:recoverytest r:core_node2) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.SolrCore [recoverytest_shard1_replica_n2] Registered new searcher Searcher@57f8ed70[recoverytest_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2821344 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572987020363431936
   [junit4]   2> 2821345 INFO  (searcherExecutor-13042-thread-1-processing-n:127.0.0.1:48835_solr x:recoverytest_shard1_replica_n1 s:shard1 c:recoverytest r:core_node1) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.SolrCore [recoverytest_shard1_replica_n1] Registered new searcher Searcher@ad5da8[recoverytest_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2821345 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572987020364480512
   [junit4]   2> 2821353 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2821353 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2821353 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/
   [junit4]   2> 2821355 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica_n1 url=http://127.0.0.1:48835/solr START replicas=[http://127.0.0.1:35947/solr/recoverytest_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 2821355 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica_n1 url=http://127.0.0.1:48835/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2821357 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.c.S.Request [recoverytest_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2821357 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2821357 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2821357 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2821366 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/ shard1
   [junit4]   2> 2821470 INFO  (zkCallback-4778-thread-2-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821470 INFO  (zkCallback-4779-thread-1-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821470 INFO  (zkCallback-4778-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821470 INFO  (zkCallback-4779-thread-2-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821474 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2821478 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1775
   [junit4]   2> 2821587 INFO  (zkCallback-4779-thread-2-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821588 INFO  (zkCallback-4778-thread-2-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821587 INFO  (zkCallback-4778-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2821588 INFO  (zkCallback-4779-thread-1-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2822360 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2659
   [junit4]   2> 2822367 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2822466 INFO  (zkCallback-4779-thread-1-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2822466 INFO  (zkCallback-4778-thread-2-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2822466 INFO  (zkCallback-4779-thread-2-processing-n:127.0.0.1:35947_solr) [n:127.0.0.1:35947_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2822466 INFO  (zkCallback-4778-thread-1-processing-n:127.0.0.1:48835_solr) [n:127.0.0.1:48835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2823368 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3986
   [junit4]   2> 2823370 INFO  (TEST-HdfsRecoveryZkTest.test-seed#[4B5C98696D8196D8]) [    ] o.a.s.c.RecoveryZkTest Indexing 700 documents
   [junit4]   2> 2823387 DEBUG (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.HdfsTransactionLog Opening new tlog hdfs tlog{file=hdfs://serv1-vm3:37218/data/recoverytest/core_node1/data/tlog/tlog.0000000000000000000 refcount=1}
   [junit4]   2> 2823390 INFO  (OverseerCollectionConfigSetProcessor-98311688488812549-127.0.0.1:48835_solr-n_0000000000) [n:127.0.0.1:48835_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2823397 DEBUG (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.HdfsTransactionLog Opening new tlog hdfs tlog{file=hdfs://serv1-vm3:37218/data/recoverytest/core_node2/data/tlog/tlog.0000000000000000000 refcount=1}
   [junit4]   2> 2823406 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-0 (1572987022494138368)]} 0 15
   [junit4]   2> 2823406 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0 (1572987022494138368)]} 0 30
   [junit4]   2> 2823406 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-0 (1572987022494138369)]} 0 16
   [junit4]   2> 2823407 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0 (1572987022494138369)]} 0 31
   [junit4]   2> 2823409 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-1 (1572987022526644224)]} 0 1
   [junit4]   2> 2823409 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1 (1572987022526644224)]} 0 2
   [junit4]   2> 2823409 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-1 (1572987022527692800)]} 0 1
   [junit4]   2> 2823410 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1 (1572987022527692800)]} 0 2
   [junit4]   2> 2823412 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-2 (1572987022529789952)]} 0 1
   [junit4]   2> 2823412 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2 (1572987022529789952)]} 0 2
   [junit4]   2> 2823412 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-2 (1572987022529789953)]} 0 0
   [junit4]   2> 2823412 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2 (1572987022529789953)]} 0 2
   [junit4]   2> 2823414 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-3 (1572987022532935681)]} 0 1
   [junit4]   2> 2823415 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-3 (1572987022532935680)]} 0 1
   [junit4]   2> 2823415 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3 (1572987022532935681)]} 0 1
   [junit4]   2> 2823415 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3 (1572987022532935680)]} 0 2
   [junit4]   2> 2823417 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-4 (1572987022536081408)]} 0 0
   [junit4]   2> 2823417 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-4 (1572987022535032832)]} 0 0
   [junit4]   2> 2823417 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4 (1572987022536081408)]} 0 1
   [junit4]   2> 2823417 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4 (1572987022535032832)]} 0 2
   [junit4]   2> 2823419 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-5 (1572987022538178560)]} 0 0
   [junit4]   2> 2823419 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-5 (1572987022538178561)]} 0 1
   [junit4]   2> 2823420 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-5 (1572987022538178560)]} 0 1
   [junit4]   2> 2823420 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5 (1572987022538178561)]} 0 1
   [junit4]   2> 2823422 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-6 (1572987022540275712)]} 0 1
   [junit4]   2> 2823422 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-6 (1572987022540275713)]} 0 0
   [junit4]   2> 2823422 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-6 (1572987022540275712)]} 0 1
   [junit4]   2> 2823422 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-6 (1572987022540275713)]} 0 2
   [junit4]   2> 2823424 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-7 (1572987022543421440)]} 0 0
   [junit4]   2> 2823425 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-7 (1572987022543421440)]} 0 1
   [junit4]   2> 2823425 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-7 (1572987022543421441)]} 0 1
   [junit4]   2> 2823425 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-7 (1572987022543421441)]} 0 2
   [junit4]   2> 2823427 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-8 (1572987022545518592)]} 0 0
   [junit4]   2> 2823427 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-8 (1572987022546567168)]} 0 0
   [junit4]   2> 2823427 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-8 (1572987022545518592)]} 0 1
   [junit4]   2> 2823427 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-8 (1572987022546567168)]} 0 1
   [junit4]   2> 2823429 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-9 (1572987022548664321)]} 0 0
   [junit4]   2> 2823430 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-9 (1572987022548664321)]} 0 1
   [junit4]   2> 2823430 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-9 (1572987022548664320)]} 0 1
   [junit4]   2> 2823430 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-9 (1572987022548664320)]} 0 2
   [junit4]   2> 2823431 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-7 (-1572987022550761472)]} 0 0
   [junit4]   2> 2823432 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-7 (-1572987022550761473)]} 0 0
   [junit4]   2> 2823432 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-7 (-1572987022550761472)]} 0 1
   [junit4]   2> 2823432 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-7 (-1572987022550761473)]} 0 1
   [junit4]   2> 2823434 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-10 (1572987022553907200)]} 0 0
   [junit4]   2> 2823434 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-10 (1572987022552858624)]} 0 1
   [junit4]   2> 2823435 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-10 (1572987022552858624)]} 0 2
   [junit4]   2> 2823436 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-10 (1572987022553907200)]} 0 2
   [junit4]   2> 2823437 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-8 (-1572987022557052928)]} 0 0
   [junit4]   2> 2823437 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-8 (-1572987022557052928)]} 0 1
   [junit4]   2> 2823438 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-8 (-1572987022557052929)]} 0 1
   [junit4]   2> 2823438 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-8 (-1572987022557052929)]} 0 1
   [junit4]   2> 2823440 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-11 (1572987022559150080)]} 0 0
   [junit4]   2> 2823440 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-11 (1572987022559150080)]} 0 1
   [junit4]   2> 2823440 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-11 (1572987022560198656)]} 0 0
   [junit4]   2> 2823441 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-11 (1572987022560198656)]} 0 2
   [junit4]   2> 2823442 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-11 (-1572987022561247232)]} 0 0
   [junit4]   2> 2823442 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-11 (-1572987022561247232)]} 0 1
   [junit4]   2> 2823443 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-11 (-1572987022562295808)]} 0 0
   [junit4]   2> 2823443 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-11 (-1572987022562295808)]} 0 1
   [junit4]   2> 2823445 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-12 (1572987022564392961)]} 0 0
   [junit4]   2> 2823445 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-12 (1572987022564392960)]} 0 1
   [junit4]   2> 2823445 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-12 (1572987022564392961)]} 0 1
   [junit4]   2> 2823445 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-12 (1572987022564392960)]} 0 2
   [junit4]   2> 2823448 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-13 (1572987022567538689)]} 0 1
   [junit4]   2> 2823448 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-13 (1572987022567538688)]} 0 1
   [junit4]   2> 2823448 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-13 (1572987022567538689)]} 0 2
   [junit4]   2> 2823448 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-13 (1572987022567538688)]} 0 2
   [junit4]   2> 2823450 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-13 (-1572987022570684416)]} 0 0
   [junit4]   2> 2823451 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-13 (-1572987022570684417)]} 0 1
   [junit4]   2> 2823451 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-13 (-1572987022570684416)]} 0 2
   [junit4]   2> 2823451 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-13 (-1572987022570684417)]} 0 2
   [junit4]   2> 2823454 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-14 (1572987022573830145)]} 0 1
   [junit4]   2> 2823455 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-14 (1572987022573830145)]} 0 2
   [junit4]   2> 2823455 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-14 (1572987022573830144)]} 0 1
   [junit4]   2> 2823455 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-14 (1572987022573830144)]} 0 3
   [junit4]   2> 2823457 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-14 (-1572987022576975872)]} 0 0
   [junit4]   2> 2823457 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-14 (-1572987022578024448)]} 0 0
   [junit4]   2> 2823457 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-14 (-1572987022576975872)]} 0 1
   [junit4]   2> 2823457 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-14 (-1572987022578024448)]} 0 1
   [junit4]   2> 2823459 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-15 (1572987022580121600)]} 0 0
   [junit4]   2> 2823459 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-15 (1572987022580121601)]} 0 1
   [junit4]   2> 2823459 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-15 (1572987022580121600)]} 0 1
   [junit4]   2> 2823460 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-15 (1572987022580121601)]} 0 1
   [junit4]   2> 2823461 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-16 (1572987022582218753)]} 0 0
   [junit4]   2> 2823461 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-16 (1572987022582218752)]} 0 0
   [junit4]   2> 2823462 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-16 (1572987022582218752)]} 0 1
   [junit4]   2> 2823462 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-16 (1572987022582218753)]} 0 1
   [junit4]   2> 2823464 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-17 (1572987022584315905)]} 0 0
   [junit4]   2> 2823464 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-17 (1572987022584315904)]} 0 0
   [junit4]   2> 2823464 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-17 (1572987022584315905)]} 0 1
   [junit4]   2> 2823464 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-17 (1572987022584315904)]} 0 2
   [junit4]   2> 2823466 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-18 (1572987022587461632)]} 0 0
   [junit4]   2> 2823466 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-18 (1572987022587461633)]} 0 0
   [junit4]   2> 2823466 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-18 (1572987022587461632)]} 0 1
   [junit4]   2> 2823466 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-18 (1572987022587461633)]} 0 1
   [junit4]   2> 2823468 INFO  (qtp481455390-32800) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-15 (-1572987022589558784)]} 0 0
   [junit4]   2> 2823468 INFO  (qtp481455390-32802) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-15 (-1572987022589558785)]} 0 0
   [junit4]   2> 2823469 INFO  (qtp222404833-32805) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-15 (-1572987022589558784)]} 0 1
   [junit4]   2> 2823469 INFO  (qtp222404833-32809) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-15 (-1572987022589558785)]} 0 1
   [junit4]   2> 2823470 INFO  (qtp481455390-32797) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-19 (1572987022591655936)]} 0 0
   [junit4]   2> 2823471 INFO  (qtp222404833-32808) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-19 (1572987022591655936)]} 0 1
   [junit4]   2> 2823474 INFO  (qtp481455390-32798) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[2-19 (1572987022591655937)]} 0 4
   [junit4]   2> 2823475 INFO  (qtp222404833-32807) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-19 (1572987022591655937)]} 0 5
   [junit4]   2> 2823476 INFO  (qtp481455390-32801) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[1-16 (-1572987022597947392)]} 0 1
   [junit4]   2> 2823476 INFO  (qtp222404833-32794) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-16 (-1572987022597947392)]} 0 1
   [junit4]   2> 2823477 INFO  (qtp481455390-32799) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{delete=[2-16 (-1572987022597947393)]} 0 0
   [junit4]   2> 2823477 INFO  (qtp222404833-32804) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-16 (-1572987022597947393)]} 0 1
   [junit4]   2> 2823479 INFO  (qtp481455390-32795) [n:127.0.0.1:35947_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:48835/solr/recoverytest_shard1_replica_n1/&wt=javabin&version=2}{add=[1-20 (1572987022600044544)]} 0 1
   [junit4]   2> 2823479 INFO  (qtp222404833-32806) [n:127.0.0.1:48835_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  webap

[...truncated too long message...]

rce, HdfsUnloadDistributedZkTest, V2ApiIntegrationTest, MoreLikeThisHandlerTest, ParsingFieldUpdateProcessorsTest, TestComponentsName, TestSchemaManager, AnalyticsMergeStrategyTest, SortSpecParsingTest, TestHdfsUpdateLog, HdfsRestartWhileUpdatingTest, TestCoreDiscovery, MigrateRouteKeyTest, DebugComponentTest, AssignTest, TestFastLRUCache, TestReloadAndDeleteDocs, SynonymTokenizerTest, AddBlockUpdateTest, TestPostingsSolrHighlighter, TestDFISimilarityFactory, OverseerStatusTest, CheckHdfsIndexTest, SparseHLLTest, OpenCloseCoreStressTest, TestTrie, ZkStateReaderTest, TestCloudInspectUtil, TestPayloadScoreQParserPlugin, ZkNodePropsTest, TestDistribDocBasedVersion, TestStressUserVersions, UUIDUpdateProcessorFallbackTest, ConnectionManagerTest, TestCollationFieldDocValues, TestLockTree, TestPartialUpdateDeduplication, DocValuesNotIndexedTest, TestTolerantUpdateProcessorRandomCloud, StatelessScriptUpdateProcessorFactoryTest, TestSubQueryTransformerCrossCore, DistributedFacetPivotLargeTest, TestJoin, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestSolrCloudWithDelegationTokens, LeaderFailoverAfterPartitionTest, TestOnReconnectListenerSupport, TestBlendedInfixSuggestions, TriLevelCompositeIdRoutingTest, TestPullReplica, TestRawTransformer, TestRTimerTree, LeaderInitiatedRecoveryOnCommitTest, TestLMJelinekMercerSimilarityFactory, TestCollectionAPIs, DocValuesMultiTest, TestFieldSortValues, TestSolrJ, BasicZkTest, TestAnalyzedSuggestions, SuggestComponentTest, TestConfigSetsAPIExclusivity, TestManagedSynonymGraphFilterFactory, TestRemoteStreaming, TestSolrCloudWithSecureImpersonation, CopyFieldTest, FullSolrCloudDistribCmdsTest, TestSortByMinMaxFunction, TestConfig, TestCloudNestedDocsSort, SolrJmxReporterTest, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, TestBackupRepositoryFactory, BlockCacheTest, BlockJoinFacetSimpleTest, TestSimpleTextCodec, TestAuthenticationFramework, OverriddenZkACLAndCredentialsProvidersTest, TestDistributedMissingSort, TestDocBasedVersionConstraints, TestCollectionAPI, HdfsThreadLeakTest, TestAuthorizationFramework, TestSolrCoreSnapshots, PolyFieldTest, BufferStoreTest, TestLRUCache, BaseCdcrDistributedZkTest, FileBasedSpellCheckerTest, TestFieldTypeCollectionResource, TestLRUStatsCache, DocumentAnalysisRequestHandlerTest, TestReloadDeadlock, CollectionsAPIDistributedZkTest, NodeMutatorTest, PingRequestHandlerTest, TestFieldCache, TestRuleBasedAuthorizationPlugin, TestClassNameShortening, BlockJoinFacetDistribTest, TermsComponentTest, TestExclusionRuleCollectionAccess, SubstringBytesRefFilterTest, TestMinMaxOnMultiValuedField, TestSolr4Spatial2, CdcrRequestHandlerTest, TestImplicitCoreProperties, TestRangeQuery, TestRandomFlRTGCloud, TestInitParams, DistributedExpandComponentTest, TestConfigSets, TestCryptoKeys, TestMergePolicyConfig, SpatialHeatmapFacetsTest, TestMaxScoreQueryParser, TestPushWriter, SolrGangliaReporterTest, TestSolrCloudWithHadoopAuthPlugin, TestMultiWordSynonyms, SaslZkACLProviderTest, TestSolrQueryParser, DistributedQueueTest, DistributedFacetPivotSmallAdvancedTest, SearchHandlerTest, DistributedTermsComponentTest, TestNamedUpdateProcessors, HdfsRecoverLeaseTest, BasicDistributedZk2Test, UnloadDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, TestReplicationHandler, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, TestRandomDVFaceting, ShardRoutingCustomTest, TestDistributedSearch, AutoCommitTest, TestDistributedGrouping, TestRecovery, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, TestStressReorder, TestReload, StatsComponentTest, SpellCheckComponentTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, TestFiltering, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestIndexSearcher, SoftAutoCommitTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTSTTest, SpatialFilterTest, SuggesterWFSTTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, QueryEqualityTest, DefaultValueUpdateProcessorTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, CacheHeaderTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, DirectSolrSpellCheckerTest, TestQueryTypes, TestOmitPositions, XmlUpdateRequestHandlerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, RequiredFieldsTest, FastVectorHighlighterTest, LoggingHandlerTest, RegexBoostProcessorTest, SolrPluginUtilsTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, QueryParsingTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, SOLR749Test, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, SolrIndexConfigTest, TestStressRecovery, SampleTest, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestCharFilters, TestXIncludeConfig, TestIBSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, PluginInfoTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, DOMUtilTest, TestDocumentBuilder, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, TestJettySolrRunner, ConnectionReuseTest, CdcrBootstrapTest, CdcrReplicationDistributedZkTest, CdcrVersionReplicationTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionStateFormat2Test, CustomCollectionTest, DeleteInactiveReplicaTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteReplicaTest, DeleteShardTest, DeleteStatusTest, DistribCursorPagingTest, DistribDocExpirationUpdateProcessorTest, DistribJoinFromCollectionTest, HttpPartitionTest, LeaderFailureAfterFreshStartTest, MoveReplicaHDFSTest, MoveReplicaTest, MultiThreadedOCPTest, RemoteQueryErrorTest, ReplaceNodeTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCloudExampleTest, TestCloudRecovery, TestLeaderElectionWithEmptyReplica, TestMiniSolrCloudClusterSSL, TestPullReplicaErrorHandling, TestRebalanceLeaders, TestSegmentSorting, TestStressCloudBlindAtomicUpdates, TestStressInPlaceUpdates, VMParamsZkACLAndCredentialsProvidersTest, AutoScalingHandlerTest, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, HdfsRecoveryZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsRecoveryZkTest -Dtests.seed=4B5C98696D8196D8 -Dtests.slow=true -Dtests.locale=pt-BR -Dtests.timezone=Pacific/Pohnpei -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | HdfsRecoveryZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog
   [junit4]    > at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
   [junit4]    > at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
   [junit4]    > at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)
   [junit4]    > at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)
   [junit4]    > at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
   [junit4]    > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4]    > at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)
   [junit4]    > at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)
   [junit4]    > at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
   [junit4]    > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)
   [junit4]    > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
   [junit4]    > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
   [junit4]    > at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
   [junit4]    > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([4B5C98696D8196D8]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:296)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [561/728 (1!)] on J1 in 35.30s, 1 test, 1 failure <<< FAILURES!

[...truncated 43863 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 42 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/42/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  org.apache.solr.update.AutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([CADDD516A7788B00:700FBA6E24566515]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
        at org.apache.solr.update.AutoCommitTest.testCommitWithin(AutoCommitTest.java:353)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:529&qt=&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
        ... 40 more


FAILED:  org.apache.solr.update.HardAutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([CADDD516A7788B00:700FBA6E24566515]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
        at org.apache.solr.update.HardAutoCommitTest.testCommitWithin(HardAutoCommitTest.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:529&qt=&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
        ... 40 more


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([CADDD516A7788B00:4289EACC0984E6F8]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at org.junit.Assert.assertFalse(Assert.java:79)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.delayedReorderingFetchesMissingUpdateFromLeaderTest(TestInPlaceUpdatesDistrib.java:915)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12001 lines...]
   [junit4] Suite: org.apache.solr.update.AutoCommitTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.AutoCommitTest_CADDD516A7788B00-001/init-core-data-001
   [junit4]   2> 1875839 WARN  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=232 numCloses=232
   [junit4]   2> 1875839 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1875840 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1875841 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 1875841 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1875870 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1875887 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 1876009 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1876050 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1876053 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1876053 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1876055 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1876081 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1876108 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1876199 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1876208 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1876209 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1876209 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1876209 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.AutoCommitTest_CADDD516A7788B00-001/init-core-data-001/]
   [junit4]   2> 1876211 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1876277 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1876284 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1876285 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1876285 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1876286 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=98.447265625, floorSegmentMB=1.6474609375, forceMergeDeletesPctAllowed=8.066297445533309, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15326136282876687
   [junit4]   2> 1876286 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c147864[collection1] main]
   [junit4]   2> 1876286 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1876289 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1876311 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1876311 INFO  (coreLoadExecutor-6632-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573015431668563968
   [junit4]   2> 1876312 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1876313 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 1876316 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1876316 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1876316 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1876319 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMaxTime
   [junit4]   2> 1876319 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1876320 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@49761067[collection1] realtime]
   [junit4]   2> 1876320 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 1
   [junit4]   2> 1876321 ERROR (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 1876322 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1876325 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1876326 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1876330 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1876330 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1876330 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1876332 INFO  (searcherExecutor-6633-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c147864[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1876360 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1876386 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1876500 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1876510 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 1876524 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1876524 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.AutoCommitTest_CADDD516A7788B00-001/init-core-data-001/]
   [junit4]   2> 1876589 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1876589 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1876590 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@78eb7cd5[collection1] main]
   [junit4]   2> 1876591 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1876591 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1876593 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1876594 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1876594 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573015431965310976
   [junit4]   2> 1876594 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1876594 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1876594 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1876594 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1876594 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1876595 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1876595 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1876596 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1876596 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1876596 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1876598 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@78eb7cd5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1876598 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1bd5f7f0[collection1] main]
   [junit4]   2> 1876598 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@45328b06
   [junit4]   2> 1876599 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1160940294
   [junit4]   2> 1876600 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1bd5f7f0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1876607 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[529 (1573015431977893888)]} 0 1
   [junit4]   2> 1876607 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1878110 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 1878113 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@633ad707[collection1] main]
   [junit4]   2> 1878114 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1878115 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@633ad707[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C1)))}
   [junit4]   2> 1878362 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[530 (1573015433818144768)]} 0 1756
   [junit4]   2> 1878362 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 1878363 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1878363 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{delete=[529 (-1573015433820241920)]} 0 0
   [junit4]   2> 1878364 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 1879864 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 1879865 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2520ef31[collection1] main]
   [junit4]   2> 1879865 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1879866 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2520ef31[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):C1)))}
   [junit4]   2> 1879867 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[550 (1573015435397300224)]} 0 3262
   [junit4]   2> 1879867 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1879868 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435398348800)]} 0 3263
   [junit4]   2> 1879868 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435398348801)]} 0 3263
   [junit4]   2> 1879868 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435398348802)]} 0 3263
   [junit4]   2> 1879868 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435398348803)]} 0 3264
   [junit4]   2> 1879868 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435398348804)]} 0 3264
   [junit4]   2> 1879869 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435399397376)]} 0 3264
   [junit4]   2> 1879869 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435399397377)]} 0 3264
   [junit4]   2> 1879869 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435399397378)]} 0 3264
   [junit4]   2> 1879869 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435399397379)]} 0 3264
   [junit4]   2> 1879869 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[500 (1573015435399397380)]} 0 3264
   [junit4]   2> 1879869 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:500&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1881368 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 1881370 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f1818db[collection1] main]
   [junit4]   2> 1881370 INFO  (commitScheduler-6641-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1881371 INFO  (searcherExecutor-6638-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7f1818db[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):C1) Uninverting(_2(7.1.0):C11/9:delGen=1)))}
   [junit4]   2> 1881381 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[531 (1573015436984844288)]} 0 4777
   [junit4]   2> 1881381 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:500&qt=&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 1881382 INFO  (TEST-AutoCommitTest.testMaxTime-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testMaxTime
   [junit4]   2> 1881385 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Starting testCommitWithin
   [junit4]   2> 1881385 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1881385 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@50d1514a[collection1] realtime]
   [junit4]   2> 1881386 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 1881386 ERROR (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 1881387 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1881416 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1881431 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1881560 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1881567 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 1881575 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1881575 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.AutoCommitTest_CADDD516A7788B00-001/init-core-data-001/]
   [junit4]   2> 1881638 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1881639 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1881639 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@68f6787f[collection1] main]
   [junit4]   2> 1881640 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1881640 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1881643 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1881643 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1881643 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1881643 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1881643 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1881643 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1881644 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1881644 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1881644 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1881645 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1881646 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@68f6787f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1881699 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1348281694, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1881700 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1881700 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7ac6d15d[collection1] main]
   [junit4]   2> 1881700 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3d08d9e1
   [junit4]   2> 1881701 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1023990241
   [junit4]   2> 1881703 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[529 (1573015437321437184)]} 0 0
   [junit4]   2> 1881703 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1881704 INFO  (searcherExecutor-6642-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7ac6d15d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1881956 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[530 (1573015437587775488)]} 0 254
   [junit4]   2> 1881957 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:530&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1881957 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {delete=[529 (-1573015437588824064)]} 0 254
   [junit4]   2> 1881957 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:529&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1881957 ERROR (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2>
   [junit4]   2> request was:q=id:529&qt=&start=0&rows=20&version=2.2
   [junit4]   2> 1881958 ERROR (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=id:529&qt=&start=0&rows=20&version=2.2:java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]   2> xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2>
   [junit4]   2> request was:q=id:529&qt=&start=0&rows=20&version=2.2
   [junit4]   2> at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
   [junit4]   2> at org.apache.solr.update.AutoCommitTest.testCommitWithin(AutoCommitTest.java:353)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 1881958 INFO  (TEST-AutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testCommitWithin
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AutoCommitTest -Dtests.method=testCommitWithin -Dtests.seed=CADDD516A7788B00 -Dtests.slow=true -Dtests.locale=ar-QA -Dtests.timezone=Indian/Christmas -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.58s J1 | AutoCommitTest.testCommitWithin <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > at __randomizedtesting.SeedInfo.seed([CADDD516A7788B00:700FBA6E24566515]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:878)
   [junit4]    > at org.apache.solr.update.AutoCommitTest.testCommitWithin(AutoCommitTest.java:353)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
   [junit4]    > xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > request was:q=id:529&qt=&start=0&rows=20&version=2.2
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:871)
   [junit4]    > ... 40 more
   [junit4]   2> 1881962 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Starting testMaxDocs
   [junit4]   2> 1881962 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1881962 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1a557ff3[collection1] realtime]
   [junit4]   2> 1881962 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 1881962 ERROR (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 1881963 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1881991 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1882004 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1882074 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1882082 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 1882090 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1882090 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.AutoCommitTest_CADDD516A7788B00-001/init-core-data-001/]
   [junit4]   2> 1882132 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1882132 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1882132 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@66f9ac9f[collection1] main]
   [junit4]   2> 1882133 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1882133 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1882136 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1882136 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1882136 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1882136 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1882136 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1882136 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1882136 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1882137 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1882137 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1882137 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1882137 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=56.47265625, floorSegmentMB=1.3369140625, forceMergeDeletesPctAllowed=1.9645510651327525, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6456142463239488
   [junit4]   2> 1882138 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1882139 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@66f9ac9f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1882139 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@38106405[collection1] main]
   [junit4]   2> 1882139 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@724fce4c
   [junit4]   2> 1882140 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1917832780
   [junit4]   2> 1882140 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@38106405[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1882144 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[0 (1573015437784907776)]} 0 0
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[1 (1573015437785956352)]} 0 0
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[2 (1573015437785956353)]} 0 0
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[3 (1573015437785956354)]} 0 0
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[4 (1573015437785956355)]} 0 0
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[5 (1573015437785956356)]} 0 1
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[6 (1573015437785956357)]} 0 1
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[7 (1573015437785956358)]} 0 1
   [junit4]   2> 1882145 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[8 (1573015437785956359)]} 0 1
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[9 (1573015437785956360)]} 0 1
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[10 (1573015437787004928)]} 0 1
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[11 (1573015437787004929)]} 0 1
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[12 (1573015437787004930)]} 0 1
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[13 (1573015437787004931)]} 0 1
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1882146 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[14 (1573015437787004932)]} 0 2
   [junit4]   2> 1882148 INFO  (commitScheduler-6649-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 1882151 INFO  (commitScheduler-6649-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@62c89901[collection1] main]
   [junit4]   2> 1882152 INFO  (searcherExecutor-6646-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@62c89901[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(7.1.0):C15)))}
   [junit4]   2> 1882152 INFO  (commitScheduler-6649-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1882397 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] {add=[15 (1573015438050197504)]} 0 253
   [junit4]   2> 1882397 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&qt=&start=0&rows=20&version=2.2} hits=1 status=0 QTime=0
   [junit4]   2> 1882398 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:15&qt=&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 1882398 INFO  (TEST-AutoCommitTest.testMaxDocs-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testMaxDocs
   [junit4]   2> 1882398 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 1882398 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    x:collection1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=536448535
   [junit4]   2> 1882398 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1882399 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1882400 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1882400 INFO  (SUITE-AutoCommitTest-seed#[CADDD516A7788B00]-worker) [    x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1882400 INFO  (coreCloseExecutor-6650-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@125d56e3
   [junit4]   2> 1882401 INFO  (coreCloseExecutor-6650-thread-1-processing-x:collection1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=308106979
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.AutoCommitTest_CADDD516A7788B00-001
   [junit4]   2> Jul 15, 2017 6:47:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field_t=Lucene50(blocksize=128), multiDefault=PostingsFormat(name=LuceneVarGapFixedInterval), subject=PostingsFormat(name=LuceneVarGapFixedInterval), id=Lucene50(blocksize=128), text=BlockTreeOrds(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=675, maxMBSortInHeap=7.138136935879366, sim=RandomSimilarity(queryNorm=false): {}, locale=ar-QA, timezone=Indian/Christmas
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=239928680,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestCrossCoreJoin, TestManagedResource, TestLeaderElectionWithEmptyReplica, TestCloudInspectUtil, TestOnReconnectListenerSupport, ParsingFieldUpdateProcessorsTest, TriLevelCompositeIdRoutingTest, TestConfigSetsAPIExclusivity, HdfsRestartWhileUpdatingTest, TestShortCircuitedRequests, VersionInfoTest, TestComponentsName, DistributedFacetPivotLargeTest, TestBlendedInfixSuggestions, TestRestManager, TestConfig, CustomCollectionTest, TestFiltering, TestMaxScoreQueryParser, DistributedQueryElevationComponentTest, CursorMarkTest, TestWordDelimiterFilterFactory, SecurityConfHandlerTest, TermVectorComponentDistributedTest, BufferStoreTest, DistributedFacetPivotWhiteBoxTest, AssignTest, DocValuesNotIndexedTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestLRUStatsCache, SolrCloudReportersTest, CollectionStateFormat2Test, TestLegacyFieldReuse, TestTolerantUpdateProcessorRandomCloud, TestRandomFlRTGCloud, TestDistributedMissingSort, PingRequestHandlerTest, TestUnifiedSolrHighlighter, TestCollectionAPI, ConcurrentDeleteAndCreateCollectionTest, PeerSyncReplicationTest, ActionThrottleTest, TestIndexSearcher, BasicFunctionalityTest, CheckHdfsIndexTest, TestReRankQParserPlugin, SuggesterTSTTest, BlockJoinFacetSimpleTest, SearchHandlerTest, LukeRequestHandlerTest, TestFieldCollectionResource, TestGraphMLResponseWriter, CdcrUpdateLogTest, TestDelegationWithHadoopAuth, HdfsBasicDistributedZk2Test, TestExportWriter, TestRTGBase, TestDistribDocBasedVersion, SolrIndexConfigTest, TestBM25SimilarityFactory, OverseerStatusTest, TestMacroExpander, ShardRoutingCustomTest, TestSuggestSpellingConverter, MigrateRouteKeyTest, TestPerFieldSimilarity, SpellPossibilityIteratorTest, DeleteStatusTest, TestSSLRandomization, TestFastOutputStream, TestManagedSynonymGraphFilterFactory, DistribDocExpirationUpdateProcessorTest, TestSolrIndexConfig, TestPostingsSolrHighlighter, TestCollationFieldDocValues, TestRecovery, TestPayloadScoreQParserPlugin, TestCloudPivotFacet, DOMUtilTest, TestReloadDeadlock, SubstringBytesRefFilterTest, TermsComponentTest, PluginInfoTest, TestFastWriter, TestExclusionRuleCollectionAccess, TestCharFilters, CloudExitableDirectoryReaderTest, HdfsDirectoryFactoryTest, ExplicitHLLTest, TestRangeQuery, TestFreeTextSuggestions, TestCloudRecovery, TestFieldTypeCollectionResource, DeleteLastCustomShardedReplicaTest, DistributedExpandComponentTest, TestAuthorizationFramework, LeaderInitiatedRecoveryOnCommitTest, DistributedFacetPivotSmallAdvancedTest, SimpleMLTQParserTest, TestXmlQParser, TestInitParams, TestSolr4Spatial2, SpellCheckCollatorWithCollapseTest, TestManagedSchema, TestReplicationHandlerBackup, FileUtilsTest, SolrMetricReporterTest, LeaderFailoverAfterPartitionTest, MergeStrategyTest, CdcrReplicationDistributedZkTest, TestElisionMultitermQuery, ReplicaListTransformerTest, TestAnalyzedSuggestions, LeaderElectionIntegrationTest, StatelessScriptUpdateProcessorFactoryTest, SolrCmdDistributorTest, TestNamedUpdateProcessors, TestSubQueryTransformerCrossCore, TestSolrCloudWithDelegationTokens, StatsComponentTest, TestLegacyTerms, SearchHandlerTest, TestBadConfig, TestLMJelinekMercerSimilarityFactory, TestSchemaManager, HdfsChaosMonkeyNothingIsSafeTest, ReplicationFactorTest, MoveReplicaHDFSTest, RemoteQueryErrorTest, SolrCoreCheckLockOnStartupTest, DistributedQueueTest, CdcrRequestHandlerTest, TestQueryUtils, DocValuesMissingTest, V2ApiIntegrationTest, JSONWriterTest, TestDocumentBuilder, PeerSyncWithIndexFingerprintCachingTest, DeleteReplicaTest, SimpleCollectionCreateDeleteTest, TestGraphTermsQParserPlugin, DistributedIntervalFacetingTest, TestCollectionAPIs, TestRemoteStreaming, DebugComponentTest, ZkNodePropsTest, ShowFileRequestHandlerTest, TestSolrJ, HLLSerializationTest, TestFieldCacheSort, TestSolrXml, TestCSVLoader, HdfsRecoverLeaseTest, SpellCheckComponentTest, TestDFISimilarityFactory, PolyFieldTest, TestHdfsUpdateLog, QueryEqualityTest, TestReload, MoveReplicaTest, NodeMutatorTest, CopyFieldTest, TestCollationField, UUIDUpdateProcessorFallbackTest, CollectionsAPIAsyncDistributedZkTest, HdfsLockFactoryTest, TestMergePolicyConfig, TestLockTree, TestStressLucene, TestFieldCache, BasicDistributedZk2Test, ConvertedLegacyTest, BasicZkTest, TestSolrConfigHandlerConcurrent, SuggestComponentTest, ManagedSchemaRoundRobinCloudTest, TestRealTimeGet, ClassificationUpdateProcessorFactoryTest, BasicDistributedZkTest, TestRandomFaceting, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, AutoCommitTest]
   [junit4] Completed [376/728 (1!)] on J1 in 6.83s, 3 tests, 1 error <<< FAILURES!

[...truncated 20 lines...]
   [junit4] Suite: org.apache.solr.update.HardAutoCommitTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.HardAutoCommitTest_CADDD516A7788B00-001/init-core-data-001
   [junit4]   2> 1898575 WARN  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=94 numCloses=94
   [junit4]   2> 1898575 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1898576 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1898576 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 1898577 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1898599 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1898613 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 1898765 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1898851 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1898853 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1898854 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1898855 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1898998 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1899050 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1899221 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1899229 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1899229 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1899229 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1899229 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.HardAutoCommitTest_CADDD516A7788B00-001/init-core-data-001/]
   [junit4]   2> 1899230 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=3.5751953125, floorSegmentMB=0.728515625, forceMergeDeletesPctAllowed=9.007812898007415, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.677121944296208
   [junit4]   2> 1899274 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1899274 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1899274 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1899274 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1899275 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=2.66796875, floorSegmentMB=1.822265625, forceMergeDeletesPctAllowed=9.557656141624363, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1899275 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@35a5c34e[collection1] main]
   [junit4]   2> 1899276 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1899280 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1899325 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1899325 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1899325 INFO  (coreLoadExecutor-6786-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573015455800492032
   [junit4]   2> 1899325 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1899325 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1899325 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1899328 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1899328 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1899329 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1899330 INFO  (SUITE-HardAutoCommitTest-seed#[CADDD516A7788B00]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 1899331 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1899331 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1899333 INFO  (searcherExecutor-6787-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@35a5c34e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1899337 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCommitWithin
   [junit4]   2> 1899337 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1899337 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@79e81a18[collection1] realtime]
   [junit4]   2> 1899338 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0
   [junit4]   2> 1899338 ERROR (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/core.properties
   [junit4]   2> 1899339 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1899401 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1899416 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1899637 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1899655 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 1899682 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@30941e24
   [junit4]   2> 1899682 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.update.HardAutoCommitTest_CADDD516A7788B00-001/init-core-data-001/]
   [junit4]   2> 1899748 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1899748 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1899749 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@46c1a4c6[collection1] main]
   [junit4]   2> 1899750 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1899750 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1899752 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1899753 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1899753 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573015456249282560
   [junit4]   2> 1899753 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1899753 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1899753 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1899753 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1899753 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1899754 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1899754 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1899755 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1348281694, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1899755 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1899755 INFO  (TEST-HardAutoCommitTest.testCommitWithin-seed#[CADDD516A7788B00]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1899757 INFO  (searcherExecutor-6792-thread-1-processing-x:collection1) [    

[...truncated too long message...]

iting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id_i=PostingsFormat(name=Asserting), title_s=PostingsFormat(name=LuceneVarGapFixedInterval), id=FSTOrd50, id_field_copy_that_does_not_support_in_place_update_s=Lucene50(blocksize=128)}, docValues:{inplace_updatable_float=DocValuesFormat(name=Lucene70), id_i=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Memory), inplace_updatable_int_with_default=DocValuesFormat(name=Lucene70), inplace_updatable_int=DocValuesFormat(name=Direct), inplace_updatable_float_with_default=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1629, maxMBSortInHeap=6.176860273648142, sim=RandomSimilarity(queryNorm=true): {}, locale=nl, timezone=Europe/Bucharest
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=3,threads=1,free=208538360,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [SolrInfoBeanTest, TestRawResponseWriter, TestDeleteCollectionOnDownNodes, SolrXmlInZkTest, TestLeaderElectionZkExpiry, PreAnalyzedUpdateProcessorTest, TestStressLiveNodes, TestCloudPseudoReturnFields, HdfsBasicDistributedZkTest, TestRandomRequestDistribution, AddSchemaFieldsUpdateProcessorFactoryTest, RankQueryTest, DistributedDebugComponentTest, TestDocTermOrds, TestSolrQueryResponse, RequestHandlersTest, DeleteInactiveReplicaTest, TestCustomSort, TestLFUCache, DistributedSpellCheckComponentTest, TestSchemalessBufferedUpdates, TestReversedWildcardFilterFactory, XmlUpdateRequestHandlerTest, TestTrackingShardHandlerFactory, TestFieldCacheWithThreads, DistributedFacetPivotLongTailTest, AsyncCallRequestStatusResponseTest, ForceLeaderTest, ChaosMonkeySafeLeaderWithPullReplicasTest, PKIAuthenticationIntegrationTest, SolrShardReporterTest, CollectionReloadTest, TestLegacyNumericUtils, TestSolrCloudWithKerberosAlt, TestMiniSolrCloudCluster, TestCursorMarkWithoutUniqueKey, TestSolrDeletionPolicy2, TestCloudJSONFacetJoinDomain, ConfigSetsAPITest, RecoveryAfterSoftCommitTest, AliasIntegrationTest, TestFilteredDocIdSet, AnalysisAfterCoreReloadTest, TestSearcherReuse, BlockDirectoryTest, TestScoreJoinQPScore, ResponseLogComponentTest, TestSolrConfigHandlerCloud, TestManagedResourceStorage, TestCSVResponseWriter, SoftAutoCommitTest, DirectSolrConnectionTest, VMParamsZkACLAndCredentialsProvidersTest, SmileWriterTest, TestQueryWrapperFilter, TestConfigSetsAPI, TestExceedMaxTermLength, TestLMDirichletSimilarityFactory, OverseerTest, TestStreamBody, ShardRoutingTest, TestHighFrequencyDictionaryFactory, MoveReplicaHDFSUlogDirTest, TestFileDictionaryLookup, DistribCursorPagingTest, TestPullReplicaErrorHandling, SimplePostToolTest, TestSchemaResource, DistribJoinFromCollectionTest, TestLazyCores, WrapperMergePolicyFactoryTest, TestCloudManagedSchema, TestSolrCoreParser, TestNoOpRegenerator, SyncSliceTest, DateMathParserTest, TestStandardQParsers, SimpleFacetsTest, HighlighterConfigTest, CachingDirectoryFactoryTest, TestObjectReleaseTracker, ClusterStateTest, TestConfigSetsAPIZkFailure, TestTolerantSearch, ClusterStateUpdateTest, AlternateDirectoryTest, FullHLLTest, TestRecoveryHdfs, TestFieldCacheReopen, TestCloudDeleteByQuery, TestBulkSchemaAPI, DeleteNodeTest, TestConfigsApi, TestSha256AuthenticationProvider, MissingSegmentRecoveryTest, DistributedFacetExistsSmallTest, TestIBSimilarityFactory, DistanceFunctionTest, TestCoreContainer, TestSurroundQueryParser, TestAnalyzeInfixSuggestions, TestCloudSchemaless, CSVRequestHandlerTest, DistanceUnitsTest, SolrMetricsIntegrationTest, LeaderInitiatedRecoveryOnShardRestartTest, TestUseDocValuesAsStored2, CdcrVersionReplicationTest, SpatialFilterTest, QueryElevationComponentTest, HdfsDirectoryTest, LeaderFailureAfterFreshStartTest, TemplateUpdateProcessorTest, TestAtomicUpdateErrorCases, TolerantUpdateProcessorTest, TestPHPSerializedResponseWriter, TestReplicaProperties, TestDistributedStatsComponentCardinality, TestMultiValuedNumericRangeQuery, SpatialRPTFieldTypeTest, TestXIncludeConfig, RAMDirectoryFactoryTest, TestSweetSpotSimilarityFactory, TestManagedSchemaAPI, ZkCLITest, ReplaceNodeTest, MultiThreadedOCPTest, RecoveryZkTest, ExitableDirectoryReaderTest, DistributedVersionInfoTest, ChaosMonkeyNothingIsSafeTest, TestPseudoReturnFields, ChangedSchemaMergeTest, TestHdfsBackupRestoreCore, TestWriterPerf, TestDistributedSearch, SolrIndexSplitterTest, DeleteShardTest, SharedFSAutoReplicaFailoverUtilsTest, TestExtendedDismaxParser, DisMaxRequestHandlerTest, InfoHandlerTest, TestFaceting, TestApiFramework, TestTolerantUpdateProcessorCloud, XsltUpdateRequestHandlerTest, TestXmlQParserPlugin, HLLUtilTest, AnalysisErrorHandlingTest, TestNRTOpen, TestSolrCloudSnapshots, TestPolicyCloud, SuggestComponentContextFilterQueryTest, TestNonDefinedSimilarityFactory, MultiTermTest, TestIndexingPerformance, UUIDFieldTest, TestSort, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, TestManagedSynonymFilterFactory, TestBinaryField, UniqFieldsUpdateProcessorFactoryTest, CollectionTooManyReplicasTest, TestNumericTerms32, TestCorePropertiesReload, TestHashPartitioner, SortByFunctionTest, TestUninvertingReader, CdcrBootstrapTest, CursorPagingTest, CdcrReplicationHandlerTest, TestBlobHandler, TestHttpShardHandlerFactory, EnumFieldTest, TestSmileRequest, ClassificationUpdateProcessorIntegrationTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, ZkControllerTest, TestStressReorder, TestStressVersions, TestSolr4Spatial, TestGroupingSearch, PeerSyncTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, CoreAdminHandlerTest, SuggesterTest, TestTrie, SuggesterWFSTTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestUpdate, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, DocValuesMultiTest, TestSolrDeletionPolicy1, CacheHeaderTest, IndexBasedSpellCheckerTest, DirectSolrSpellCheckerTest, TestQueryTypes, TestOmitPositions, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, LoggingHandlerTest, SolrPluginUtilsTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, TestQuerySenderListener, UpdateParamsTest, TestStressRecovery, TestSolrCoreProperties, TestPhraseSuggestions, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TimeZoneUtilsTest, ScriptEngineTest, URLClassifyProcessorTest, PreAnalyzedFieldTest, PrimUtilsTest, SpellingQueryConverterTest, TestLRUCache, SliceStateTest, DistributedMLTComponentTest, TestSimpleTrackingShardHandler, CollectionsAPISolrJTest, LeaderElectionContextKeyTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerTaskQueueTest, RestartWhileUpdatingTest, RollingRestartTest, SSLMigrationTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestAuthenticationFramework, TestClusterProperties, TestCryptoKeys, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore, TestLeaderInitiatedRecoveryThread, TestLocalFSCloudBackupRestore, TestMiniSolrCloudClusterSSL, TestPrepRecovery, TestRebalanceLeaders, TestRequestForwarding, TestSegmentSorting, TestSizeLimitedDistributedMap, TestTlogReplica, TlogReplayBufferedWhileIndexingTest, AutoScalingHandlerTest, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest, HdfsNNFailoverTest, HdfsSyncSliceTest, HdfsThreadLeakTest, TestClusterStateMutator, ZkStateReaderTest, ZkStateWriterTest, ImplicitSnitchTest, RuleEngineTest, RulesTest, BlobRepositoryCloudTest, ConfigureRecoveryStrategyTest, CoreSorterTest, DirectoryFactoryTest, QueryResultKeyTest, TestBackupRepositoryFactory, TestConfigOverlay, TestConfigSetImmutable, TestConfigSetProperties, TestCustomStream, TestImplicitCoreProperties, TestInfoStreamLogging, TestReloadAndDeleteDocs, TestShardHandlerFactory, TestSimpleTextCodec, TestSolrConfigHandler, RequestLoggingTest, TestConfigReload, TestSQLHandlerNonCloud, TestSystemCollAutoCreate, V2StandaloneTest, CoreAdminCreateDiscoverTest, CoreAdminRequestStatusTest, CoreMergeIndexesAdminHandlerTest, MetricsHandlerTest, PropertiesRequestHandlerTest, SegmentsInfoRequestHandlerTest, StatsReloadRaceTest, TestCoreAdminApis, DistributedFacetPivotSmallTest, DistributedSuggestComponentTest, ShufflingReplicaListTransformerTest, SpatialHeatmapFacetsTest, TestIntervalFaceting, TestMacros, TestBinaryResponseWriter, TestGeoJSONResponseWriter, TestRawTransformer, TestSubQueryTransformer, TestClassNameShortening, TestCopyFieldCollectionResource, TestDynamicFieldCollectionResource, TestDynamicFieldResource, TestFieldResource, TestFieldTypeResource, TestSchemaNameResource, TestSchemaSimilarityResource, TestSchemaVersionResource, TestSerializedLuceneMatchVersion, TestUniqueKeyFieldResource, TestManagedStopFilterFactory, BadCopyFieldTest, BooleanFieldTest, CurrencyFieldTypeTest, TestBulkSchemaConcurrent, TestUseDocValuesAsStored, AnalyticsMergeStrategyTest, TestJsonFacetRefinement, TestMinMaxOnMultiValuedField, TestOrdValues, TestSortByMinMaxFunction, BJQParserTest, BlockJoinFacetDistribTest, BlockJoinFacetRandomTest, GraphQueryTest, TestCloudNestedDocsSort, TestNestedDocsSort, TestScoreJoinQPNoScore, TestJsonRequest, CloudMLTQParserTest, TestClassicSimilarityFactory, TestPerFieldSimilarityWithDefaultOverride, TestDefaultStatsCache, TestDistribIDF, BasicAuthStandaloneTest, TestPKIAuthenticationPlugin, TestRuleBasedAuthorizationPlugin, TestImpersonationWithHadoopAuth, TestSolrCloudWithHadoopAuthPlugin, TestLegacyFieldCache, TestNumericTerms64, TestDocBasedVersionConstraints, TestInPlaceUpdatesDistrib]
   [junit4] Completed [728/728 (3!)] on J0 in 119.94s, 1 test, 1 failure <<< FAILURES!

[...truncated 43330 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Loading...