[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 374 - Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 374 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/374/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
        at __randomizedtesting.SeedInfo.seed([69FA9AFCB9C0A6AA:E1AEA526173CCB52]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SyncSliceTest.waitTillAllNodesActive(SyncSliceTest.java:235)
        at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:164)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13478 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsSyncSliceTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/init-core-data-001
   [junit4]   2> 3372049 WARN  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
   [junit4]   2> 3372050 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 3372051 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3372051 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /dhmq/j
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3372076 WARN  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3372080 WARN  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3372081 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3372091 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_47769_hdfs____.avepvj/webapp
   [junit4]   2> 3372383 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47769
   [junit4]   2> 3372447 WARN  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3372448 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3372457 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_41720_datanode____kfr1mn/webapp
   [junit4]   2> 3372818 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41720
   [junit4]   2> 3372869 WARN  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3372870 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3372882 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53322_datanode____.cgxdfo/webapp
   [junit4]   2> 3372910 INFO  (IPC Server handler 3 on 60300) [    ] BlockStateChange BLOCK* processReport: from storage DS-5c7462f8-3527-4104-8fbd-1668df93c48e node DatanodeRegistration(127.0.0.1:42709, datanodeUuid=5b31e702-5e09-4f84-b054-23f02992ea55, infoPort=56159, infoSecurePort=0, ipcPort=51558, storageInfo=lv=-56;cid=testClusterID;nsid=1446706397;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3372910 INFO  (IPC Server handler 3 on 60300) [    ] BlockStateChange BLOCK* processReport: from storage DS-18c1b5a8-238c-433d-82d7-f7da8a6f0975 node DatanodeRegistration(127.0.0.1:42709, datanodeUuid=5b31e702-5e09-4f84-b054-23f02992ea55, infoPort=56159, infoSecurePort=0, ipcPort=51558, storageInfo=lv=-56;cid=testClusterID;nsid=1446706397;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3373262 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53322
   [junit4]   2> 3373327 INFO  (IPC Server handler 8 on 60300) [    ] BlockStateChange BLOCK* processReport: from storage DS-502a01ab-e12a-40b3-8989-e8294a0e1db6 node DatanodeRegistration(127.0.0.1:51924, datanodeUuid=69dd14c5-adda-4d93-b1b7-8f06e82ce487, infoPort=60791, infoSecurePort=0, ipcPort=44395, storageInfo=lv=-56;cid=testClusterID;nsid=1446706397;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3373327 INFO  (IPC Server handler 8 on 60300) [    ] BlockStateChange BLOCK* processReport: from storage DS-17c64d68-225e-49c3-9522-601a37de701f node DatanodeRegistration(127.0.0.1:51924, datanodeUuid=69dd14c5-adda-4d93-b1b7-8f06e82ce487, infoPort=60791, infoSecurePort=0, ipcPort=44395, storageInfo=lv=-56;cid=testClusterID;nsid=1446706397;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3373412 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3373413 INFO  (Thread-19875) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3373413 INFO  (Thread-19875) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3373414 ERROR (Thread-19875) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3373513 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ZkTestServer start zk server on port:57677
   [junit4]   2> 3373521 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3373522 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3373523 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3373524 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3373525 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3373526 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3373527 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3373528 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3373529 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3373530 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3373530 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3373601 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/control-001/cores/collection1
   [junit4]   2> 3373602 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3373603 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@394b81e4{/dhmq/j,null,AVAILABLE}
   [junit4]   2> 3373603 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5f220dd2{HTTP/1.1,[http/1.1]}{127.0.0.1:41553}
   [junit4]   2> 3373603 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server Started @3375809ms
   [junit4]   2> 3373604 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:60300/hdfs__localhost_60300__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001_tempDir-002_control_data, hostContext=/dhmq/j, hostPort=41553, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/control-001/cores}
   [junit4]   2> 3373604 ERROR (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3373604 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3373604 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3373604 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3373604 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T12:07:10.909Z
   [junit4]   2> 3373606 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3373606 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/control-001/solr.xml
   [junit4]   2> 3373610 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3373611 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57677/solr
   [junit4]   2> 3373623 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3373623 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41553_dhmq%2Fj
   [junit4]   2> 3373624 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.Overseer Overseer (id=98159010166800388-127.0.0.1:41553_dhmq%2Fj-n_0000000000) starting
   [junit4]   2> 3373628 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41553_dhmq%2Fj
   [junit4]   2> 3373631 INFO  (zkCallback-3642-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3373702 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/control-001/cores
   [junit4]   2> 3373702 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3373703 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3373707 INFO  (OverseerStateUpdate-98159010166800388-127.0.0.1:41553_dhmq%2Fj-n_0000000000) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3374717 WARN  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3374718 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3374729 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3374826 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3374834 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3374834 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:60300/solr_hdfs_home
   [junit4]   2> 3374834 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3374834 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3374834 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/control-001/cores/collection1], dataDir=[hdfs://localhost:60300/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3374834 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e6812b3
   [junit4]   2> 3374835 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 3374840 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3374840 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3374843 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3374844 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3374855 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3374860 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3374860 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3374862 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3374863 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=38.8740234375, floorSegmentMB=0.5380859375, forceMergeDeletesPctAllowed=16.751568752094393, segmentsPerTier=11.0, maxCFSSegmentSizeMB=0.775390625, noCFSRatio=0.6665077455786074
   [junit4]   2> 3374872 INFO  (IPC Server handler 1 on 60300) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42709 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-502a01ab-e12a-40b3-8989-e8294a0e1db6:NORMAL:127.0.0.1:51924|RBW], ReplicaUC[[DISK]DS-5c7462f8-3527-4104-8fbd-1668df93c48e:NORMAL:127.0.0.1:42709|FINALIZED]]} size 0
   [junit4]   2> 3374873 INFO  (IPC Server handler 2 on 60300) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51924 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5c7462f8-3527-4104-8fbd-1668df93c48e:NORMAL:127.0.0.1:42709|FINALIZED], ReplicaUC[[DISK]DS-17c64d68-225e-49c3-9522-601a37de701f:NORMAL:127.0.0.1:51924|FINALIZED]]} size 0
   [junit4]   2> 3374877 WARN  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3374905 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3374905 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3374905 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3374913 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3374913 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3374913 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.2626953125, noCFSRatio=0.7276843440127573]
   [junit4]   2> 3374916 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1ed71ccc[collection1] main]
   [junit4]   2> 3374917 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3374917 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3374917 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3374919 INFO  (searcherExecutor-9445-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1ed71ccc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3374919 INFO  (coreLoadExecutor-9444-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570544164246913024
   [junit4]   2> 3374924 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3374924 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3374924 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41553/dhmq/j/collection1/
   [junit4]   2> 3374924 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3374924 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:41553/dhmq/j/collection1/ has no replicas
   [junit4]   2> 3374924 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3374927 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41553/dhmq/j/collection1/ shard1
   [junit4]   2> 3375077 INFO  (coreZkRegister-9437-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj x:collection1 c:control_collection) [n:127.0.0.1:41553_dhmq%2Fj c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3375211 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3375212 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:57677/solr ready
   [junit4]   2> 3375212 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3375212 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 3375286 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-1-001/cores/collection1
   [junit4]   2> 3375286 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-1-001
   [junit4]   2> 3375287 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3375288 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2772b9fc{/dhmq/j,null,AVAILABLE}
   [junit4]   2> 3375288 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@150d4ab4{HTTP/1.1,[http/1.1]}{127.0.0.1:46409}
   [junit4]   2> 3375288 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server Started @3377494ms
   [junit4]   2> 3375289 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:60300/hdfs__localhost_60300__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/dhmq/j, hostPort=46409, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-1-001/cores}
   [junit4]   2> 3375289 ERROR (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3375289 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3375289 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3375289 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3375289 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T12:07:12.594Z
   [junit4]   2> 3375291 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3375291 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-1-001/solr.xml
   [junit4]   2> 3375307 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3375308 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57677/solr
   [junit4]   2> 3375314 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3375315 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3375317 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46409_dhmq%2Fj
   [junit4]   2> 3375318 INFO  (zkCallback-3646-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3375318 INFO  (zkCallback-3642-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3375323 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3375358 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-1-001/cores
   [junit4]   2> 3375358 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3375359 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3375360 INFO  (OverseerStateUpdate-98159010166800388-127.0.0.1:41553_dhmq%2Fj-n_0000000000) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3375462 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3376371 WARN  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3376372 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3376384 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3376489 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3376503 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3376504 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:60300/solr_hdfs_home
   [junit4]   2> 3376504 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3376504 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3376504 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:60300/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 3376504 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e6812b3
   [junit4]   2> 3376505 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 3376514 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3376514 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3376516 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3376516 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 3376532 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 3376538 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3376538 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3376539 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3376540 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=38.8740234375, floorSegmentMB=0.5380859375, forceMergeDeletesPctAllowed=16.751568752094393, segmentsPerTier=11.0, maxCFSSegmentSizeMB=0.775390625, noCFSRatio=0.6665077455786074
   [junit4]   2> 3376549 INFO  (IPC Server handler 9 on 60300) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42709 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-502a01ab-e12a-40b3-8989-e8294a0e1db6:NORMAL:127.0.0.1:51924|RBW], ReplicaUC[[DISK]DS-18c1b5a8-238c-433d-82d7-f7da8a6f0975:NORMAL:127.0.0.1:42709|FINALIZED]]} size 0
   [junit4]   2> 3376549 INFO  (IPC Server handler 0 on 60300) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51924 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-502a01ab-e12a-40b3-8989-e8294a0e1db6:NORMAL:127.0.0.1:51924|RBW], ReplicaUC[[DISK]DS-18c1b5a8-238c-433d-82d7-f7da8a6f0975:NORMAL:127.0.0.1:42709|FINALIZED]]} size 0
   [junit4]   2> 3376555 WARN  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3376586 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3376586 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3376586 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3376594 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3376594 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3376595 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.2626953125, noCFSRatio=0.7276843440127573]
   [junit4]   2> 3376598 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@49d25886[collection1] main]
   [junit4]   2> 3376599 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3376600 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3376600 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3376601 INFO  (searcherExecutor-9456-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@49d25886[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3376602 INFO  (coreLoadExecutor-9455-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570544166011666432
   [junit4]   2> 3376607 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3376607 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3376608 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46409/dhmq/j/collection1/
   [junit4]   2> 3376608 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3376608 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46409/dhmq/j/collection1/ has no replicas
   [junit4]   2> 3376608 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3376610 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46409/dhmq/j/collection1/ shard1
   [junit4]   2> 3376712 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3376761 INFO  (coreZkRegister-9450-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3376871 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3376938 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-2-001/cores/collection1
   [junit4]   2> 3376938 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-2-001
   [junit4]   2> 3376939 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3376940 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ec1fafa{/dhmq/j,null,AVAILABLE}
   [junit4]   2> 3376940 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4c3a2433{HTTP/1.1,[http/1.1]}{127.0.0.1:37235}
   [junit4]   2> 3376941 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server Started @3379146ms
   [junit4]   2> 3376941 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:60300/hdfs__localhost_60300__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/dhmq/j, hostPort=37235, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-2-001/cores}
   [junit4]   2> 3376941 ERROR (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3376941 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3376941 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3376942 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3376942 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T12:07:14.247Z
   [junit4]   2> 3376943 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3376943 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-2-001/solr.xml
   [junit4]   2> 3376948 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3376949 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57677/solr
   [junit4]   2> 3376955 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3376956 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3376957 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37235_dhmq%2Fj
   [junit4]   2> 3376958 INFO  (zkCallback-3646-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3376958 INFO  (zkCallback-3658-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3376959 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3376963 INFO  (zkCallback-3642-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3377011 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-2-001/cores
   [junit4]   2> 3377011 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3377012 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3377013 INFO  (OverseerStateUpdate-98159010166800388-127.0.0.1:41553_dhmq%2Fj-n_0000000000) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3377114 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3377114 INFO  (zkCallback-3658-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3378023 WARN  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3378024 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3378035 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3378137 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3378145 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3378146 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:60300/solr_hdfs_home
   [junit4]   2> 3378146 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3378146 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3378146 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:60300/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 3378146 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e6812b3
   [junit4]   2> 3378146 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 3378152 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3378152 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3378154 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3378155 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 3378166 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 3378170 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3378170 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3378172 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3378172 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=38.8740234375, floorSegmentMB=0.5380859375, forceMergeDeletesPctAllowed=16.751568752094393, segmentsPerTier=11.0, maxCFSSegmentSizeMB=0.775390625, noCFSRatio=0.6665077455786074
   [junit4]   2> 3378178 INFO  (IPC Server handler 9 on 60300) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42709 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5c7462f8-3527-4104-8fbd-1668df93c48e:NORMAL:127.0.0.1:42709|RBW], ReplicaUC[[DISK]DS-17c64d68-225e-49c3-9522-601a37de701f:NORMAL:127.0.0.1:51924|RBW]]} size 0
   [junit4]   2> 3378178 INFO  (IPC Server handler 8 on 60300) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51924 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5c7462f8-3527-4104-8fbd-1668df93c48e:NORMAL:127.0.0.1:42709|RBW], ReplicaUC[[DISK]DS-17c64d68-225e-49c3-9522-601a37de701f:NORMAL:127.0.0.1:51924|RBW]]} size 0
   [junit4]   2> 3378183 WARN  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3378220 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3378220 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3378220 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3378229 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3378230 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3378231 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.2626953125, noCFSRatio=0.7276843440127573]
   [junit4]   2> 3378234 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@25d7918b[collection1] main]
   [junit4]   2> 3378235 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3378235 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3378235 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3378237 INFO  (searcherExecutor-9467-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@25d7918b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3378239 INFO  (coreLoadExecutor-9466-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570544167728185344
   [junit4]   2> 3378242 INFO  (coreZkRegister-9461-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 c:collection1) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 3378242 INFO  (updateExecutor-3655-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 3378242 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 3378243 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 3378243 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 3378243 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 3378243 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:46409/dhmq/j/collection1/] and I am [http://127.0.0.1:37235/dhmq/j/collection1/]
   [junit4]   2> 3378245 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:46409/dhmq/j]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:37235_dhmq%252Fj&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 3378246 INFO  (qtp1192816117-37056) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 3378246 INFO  (qtp1192816117-37056) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 3378246 INFO  (qtp1192816117-37056) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:37235_dhmq%2Fj, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:37235/dhmq/j","node_name":"127.0.0.1:37235_dhmq%2Fj","state":"down"}
   [junit4]   2> 3378346 INFO  (zkCallback-3658-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3378346 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3378590 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-3-001/cores/collection1
   [junit4]   2> 3378590 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-3-001
   [junit4]   2> 3378591 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3378592 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@83c963{/dhmq/j,null,AVAILABLE}
   [junit4]   2> 3378592 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4ab0f19f{HTTP/1.1,[http/1.1]}{127.0.0.1:36603}
   [junit4]   2> 3378592 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.e.j.s.Server Started @3380798ms
   [junit4]   2> 3378593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:60300/hdfs__localhost_60300__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001_tempDir-002_jetty3, solrconfig=solrconfig.xml, hostContext=/dhmq/j, hostPort=36603, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-3-001/cores}
   [junit4]   2> 3378593 ERROR (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3378593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3378593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3378593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3378593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-18T12:07:15.898Z
   [junit4]   2> 3378599 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3378600 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-3-001/solr.xml
   [junit4]   2> 3378605 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3378605 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57677/solr
   [junit4]   2> 3378611 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3378612 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3378614 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36603_dhmq%2Fj
   [junit4]   2> 3378615 INFO  (zkCallback-3658-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3378615 INFO  (zkCallback-3642-thread-1-processing-n:127.0.0.1:41553_dhmq%2Fj) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3378615 INFO  (zkCallback-3646-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3378615 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3378619 INFO  (zkCallback-3665-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3378656 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-3-001/cores
   [junit4]   2> 3378656 INFO  (TEST-HdfsSyncSliceTest.test-seed#[69FA9AFCB9C0A6AA]) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3378657 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3378660 INFO  (OverseerStateUpdate-98159010166800388-127.0.0.1:41553_dhmq%2Fj-n_0000000000) [n:127.0.0.1:41553_dhmq%2Fj    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3378761 INFO  (zkCallback-3652-thread-1-processing-n:127.0.0.1:46409_dhmq%2Fj) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3378761 INFO  (zkCallback-3658-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj) [n:127.0.0.1:37235_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3378761 INFO  (zkCallback-3665-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3379246 INFO  (qtp1192816117-37056) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:37235_dhmq%2Fj, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","dataDir":"hdfs://localhost:60300/solr_hdfs_home/collection1/core_node2/data/","base_url":"http://127.0.0.1:37235/dhmq/j","node_name":"127.0.0.1:37235_dhmq%2Fj","state":"recovering","ulogDir":"hdfs://localhost:60300/solr_hdfs_home/collection1/core_node2/data/tlog"}
   [junit4]   2> 3379246 INFO  (qtp1192816117-37056) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 3379247 INFO  (qtp1192816117-37056) [n:127.0.0.1:46409_dhmq%2Fj    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:37235_dhmq%252Fj&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
   [junit4]   2> 3379675 WARN  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3379676 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3379687 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3379747 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:46409/dhmq/j/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 3379747 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:37235/dhmq/j START replicas=[http://127.0.0.1:46409/dhmq/j/collection1/] nUpdates=100
   [junit4]   2> 3379750 INFO  (qtp1192816117-37060) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3379750 INFO  (qtp1192816117-37060) [n:127.0.0.1:46409_dhmq%2Fj c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/dhmq/j path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3379751 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:1.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3379751 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 3379751 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3379751 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 3379752 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3379752 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 3379752 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 3379752 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 3379752 INFO  (recoveryExecutor-3656-thread-1-processing-n:127.0.0.1:37235_dhmq%2Fj x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37235_dhmq%2Fj c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 3379781 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3379790 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3379791 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:60300/solr_hdfs_home
   [junit4]   2> 3379791 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3379791 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3379791 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001/shard-3-001/cores/collection1], dataDir=[hdfs://localhost:60300/solr_hdfs_home/collection1/core_node3/data/]
   [junit4]   2> 3379791 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e6812b3
   [junit4]   2> 3379792 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:60300/solr_hdfs_home/collection1/core_node3/data/snapshot_metadata
   [junit4]   2> 3379797 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3379797 INFO  (coreLoadExecutor-9477-thread-1-processing-n:127.0.0.1:36603_dhmq%2Fj) [n:127.0.0.1:36603_dhmq%2Fj c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3379803 INFO  (coreLoadExecutor-9477-t

[...truncated too long message...]

eans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:203)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.teardownClass(HdfsSyncSliceTest.java:46)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 53 more
   [junit4]   2> 3639072 INFO  (SUITE-HdfsSyncSliceTest-seed#[69FA9AFCB9C0A6AA]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_69FA9AFCB9C0A6AA-001
   [junit4]   2> Jun 18, 2017 12:11:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=ar-AE, timezone=Asia/Khandyga
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=183028920,total=526385152
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedExpandComponentTest, HdfsTlogReplayBufferedWhileIndexingTest, TestDistributedMissingSort, TestCollationFieldDocValues, TestFieldResource, RequestHandlersTest, TestConfigsApi, TestAnalyzeInfixSuggestions, BlockDirectoryTest, TestSubQueryTransformer, TestSolrDeletionPolicy1, OverseerTest, GraphQueryTest, AtomicUpdateProcessorFactoryTest, PingRequestHandlerTest, RankQueryTest, UpdateParamsTest, TestAnalyzedSuggestions, SolrIndexMetricsTest, DOMUtilTest, PrimUtilsTest, PKIAuthenticationIntegrationTest, CdcrUpdateLogTest, TestManagedSynonymFilterFactory, TestStressReorder, FieldMutatingUpdateProcessorTest, TestMaxScoreQueryParser, TestSolrCloudWithKerberosAlt, TestStressInPlaceUpdates, TestSmileRequest, PrimitiveFieldTypeTest, DirectoryFactoryTest, DocValuesMultiTest, TestDFISimilarityFactory, SolrCoreCheckLockOnStartupTest, CollectionTooManyReplicasTest, TestReloadDeadlock, MigrateRouteKeyTest, ConcurrentDeleteAndCreateCollectionTest, CurrencyFieldXmlFileTest, TestSchemaVersionResource, DeleteLastCustomShardedReplicaTest, TestExpandComponent, DistanceUnitsTest, BadComponentTest, TestCloudInspectUtil, DistributedFacetPivotLongTailTest, TestLMJelinekMercerSimilarityFactory, HttpPartitionTest, SolrCoreMetricManagerTest, TestImpersonationWithHadoopAuth, TestCollectionAPI, BooleanFieldTest, BasicDistributedZk2Test, ResponseHeaderTest, ShowFileRequestHandlerTest, StatsReloadRaceTest, ExplicitHLLTest, MinimalSchemaTest, RulesTest, TestUtils, TestGroupingSearch, TestBinaryField, TestDynamicFieldCollectionResource, TestRestoreCore, TimeZoneUtilsTest, TestPhraseSuggestions, JSONWriterTest, LeaderFailureAfterFreshStartTest, TestCorePropertiesReload, CachingDirectoryFactoryTest, DirectUpdateHandlerOptimizeTest, CdcrVersionReplicationTest, TestCollationField, TestImplicitCoreProperties, TestSolrQueryParserResource, TestReload, TestSort, TestHdfsBackupRestoreCore, TestOnReconnectListenerSupport, TestSolrFieldCacheMBean, CloudExitableDirectoryReaderTest, MultiThreadedOCPTest, TestZkChroot, DocExpirationUpdateProcessorFactoryTest, CoreAdminHandlerTest, TestXIncludeConfig, TestLFUCache, StressHdfsTest, ResponseLogComponentTest, TestGraphTermsQParserPlugin, SolrPluginUtilsTest, TestRawTransformer, HardAutoCommitTest, OpenExchangeRatesOrgProviderTest, TestNamedUpdateProcessors, TestWriterPerf, TestJmxMonitoredMap, SmileWriterTest, CursorMarkTest, QueryResultKeyTest, TestObjectReleaseTracker, TestOrdValues, TestEmbeddedSolrServerSchemaAPI, HdfsLockFactoryTest, SuggestComponentContextFilterQueryTest, TestNRTOpen, SpellPossibilityIteratorTest, CheckHdfsIndexTest, TestTestInjection, ZkStateWriterTest, TestReversedWildcardFilterFactory, TestExtendedDismaxParser, BadIndexSchemaTest, V2ApiIntegrationTest, OverseerRolesTest, HdfsRestartWhileUpdatingTest, SolrMetricsIntegrationTest, UnloadDistributedZkTest, OpenCloseCoreStressTest, ShardRoutingCustomTest, AutoCommitTest, TestDistributedGrouping, TestFaceting, TestRecovery, ZkControllerTest, TestStressVersions, TestRangeQuery, QueryElevationComponentTest, ConvertedLegacyTest, TestFiltering, TestIndexSearcher, SignatureUpdateProcessorFactoryTest, DocValuesTest, SortByFunctionTest, DistanceFunctionTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, XmlUpdateRequestHandlerTest, TestValueSourceCache, TestIndexingPerformance, RequiredFieldsTest, TestSolrQueryParser, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, QueryParsingTest, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, HighlighterConfigTest, SOLR749Test, AlternateDirectoryTest, TestQuerySenderNoQuery, TestStressRecovery, TestCharFilters, TestPerFieldSimilarity, TestFastOutputStream, PluginInfoTest, PreAnalyzedFieldTest, SpellingQueryConverterTest, TestSolrJ, TestDocumentBuilder, CircularListTest, DistributedMLTComponentTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestHighlightDedupGrouping, TestTolerantSearch, TestPathTrie, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ActionThrottleTest, AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, CdcrReplicationDistributedZkTest, CleanupOldIndexTest, CollectionReloadTest, CollectionStateFormat2Test, CollectionsAPISolrJTest, DeleteStatusTest, DistributedQueueTest, DistributedVersionInfoTest, MissingSegmentRecoveryTest, MoveReplicaTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, PeerSyncReplicationTest, RemoteQueryErrorTest, ReplaceNodeTest, ReplicationFactorTest, RollingRestartTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsNNFailoverTest, HdfsRecoveryZkTest, HdfsSyncSliceTest]
   [junit4] Completed [638/715 (1!)] on J2 in 276.91s, 1 test, 1 failure <<< FAILURES!

[...truncated 43695 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 375 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/375/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:55311/nst/fw

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:55311/nst/fw
        at __randomizedtesting.SeedInfo.seed([2C5F898E344E9D7D:A40BB6549AB2F085]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:605)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:477)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:407)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:497)
        ... 51 more




Build Log:
[...truncated 13317 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/init-core-data-001
   [junit4]   2> 3197365 WARN  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=41 numCloses=41
   [junit4]   2> 3197365 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 3197367 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3197367 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /nst/fw
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3197456 WARN  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3197463 WARN  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3197465 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3197475 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_45016_hdfs____eorwm6/webapp
   [junit4]   2> 3197825 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45016
   [junit4]   2> 3197963 WARN  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3197964 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3197977 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_55427_datanode____.sopkg0/webapp
   [junit4]   2> 3198304 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55427
   [junit4]   2> 3198362 WARN  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3198371 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3198382 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_43014_datanode____v2izx7/webapp
   [junit4]   2> 3198506 INFO  (IPC Server handler 4 on 36966) [    ] BlockStateChange BLOCK* processReport: from storage DS-96f0454d-71a1-441a-bb20-648eff417ff3 node DatanodeRegistration(127.0.0.1:33736, datanodeUuid=ac1db4d3-3667-4d48-bdee-b3b43daf786f, infoPort=34285, infoSecurePort=0, ipcPort=50116, storageInfo=lv=-56;cid=testClusterID;nsid=1185290609;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3198506 INFO  (IPC Server handler 4 on 36966) [    ] BlockStateChange BLOCK* processReport: from storage DS-35bfeff3-61da-4bcf-b500-98d455d027e8 node DatanodeRegistration(127.0.0.1:33736, datanodeUuid=ac1db4d3-3667-4d48-bdee-b3b43daf786f, infoPort=34285, infoSecurePort=0, ipcPort=50116, storageInfo=lv=-56;cid=testClusterID;nsid=1185290609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3198765 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43014
   [junit4]   2> 3198894 INFO  (IPC Server handler 3 on 36966) [    ] BlockStateChange BLOCK* processReport: from storage DS-43345fcd-5e09-4aa7-925b-a6e9805c0eaf node DatanodeRegistration(127.0.0.1:32931, datanodeUuid=e2ca1cf6-caa1-493a-ac43-2327149a533d, infoPort=37147, infoSecurePort=0, ipcPort=58714, storageInfo=lv=-56;cid=testClusterID;nsid=1185290609;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3198894 INFO  (IPC Server handler 3 on 36966) [    ] BlockStateChange BLOCK* processReport: from storage DS-5284bd1e-8b3d-4e0a-9186-4be1c3da4cd2 node DatanodeRegistration(127.0.0.1:32931, datanodeUuid=e2ca1cf6-caa1-493a-ac43-2327149a533d, infoPort=37147, infoSecurePort=0, ipcPort=58714, storageInfo=lv=-56;cid=testClusterID;nsid=1185290609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3198936 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3198936 INFO  (Thread-11045) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3198936 INFO  (Thread-11045) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3198938 ERROR (Thread-11045) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3199036 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ZkTestServer start zk server on port:58917
   [junit4]   2> 3199048 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3199049 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3199051 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3199052 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3199054 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3199055 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3199056 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3199058 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3199059 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3199060 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3199062 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3199156 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/control-001/cores/collection1
   [junit4]   2> 3199157 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@39d78eea{/nst/fw,null,AVAILABLE}
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@550e3866{HTTP/1.1,[http/1.1]}{127.0.0.1:55311}
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server Started @3201243ms
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:36966/hdfs__localhost_36966__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001_tempDir-002_control_data, hostContext=/nst/fw, hostPort=55311, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/control-001/cores}
   [junit4]   2> 3199159 ERROR (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3199159 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3199160 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T11:52:59.227Z
   [junit4]   2> 3199164 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3199164 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/control-001/solr.xml
   [junit4]   2> 3199169 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3199171 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58917/solr
   [junit4]   2> 3199191 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3199192 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55311_nst%2Ffw
   [junit4]   2> 3199193 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.Overseer Overseer (id=98164616659206148-127.0.0.1:55311_nst%2Ffw-n_0000000000) starting
   [junit4]   2> 3199202 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55311_nst%2Ffw
   [junit4]   2> 3199204 INFO  (zkCallback-31827-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3199374 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/control-001/cores
   [junit4]   2> 3199374 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3199375 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3199387 INFO  (OverseerStateUpdate-98164616659206148-127.0.0.1:55311_nst%2Ffw-n_0000000000) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3200397 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3200410 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3200509 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3200517 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3200517 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:36966/solr_hdfs_home
   [junit4]   2> 3200517 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3200517 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3200517 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/control-001/cores/collection1], dataDir=[hdfs://localhost:36966/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3200518 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78d9b800
   [junit4]   2> 3200518 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 3200524 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3200524 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3200524 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3200531 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3200533 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3200545 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3200550 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3200550 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3200550 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3200556 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3200557 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3200567 INFO  (IPC Server handler 4 on 36966) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33736 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-43345fcd-5e09-4aa7-925b-a6e9805c0eaf:NORMAL:127.0.0.1:32931|RBW], ReplicaUC[[DISK]DS-96f0454d-71a1-441a-bb20-648eff417ff3:NORMAL:127.0.0.1:33736|RBW]]} size 0
   [junit4]   2> 3200568 INFO  (IPC Server handler 6 on 36966) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32931 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-96f0454d-71a1-441a-bb20-648eff417ff3:NORMAL:127.0.0.1:33736|RBW], ReplicaUC[[DISK]DS-5284bd1e-8b3d-4e0a-9186-4be1c3da4cd2:NORMAL:127.0.0.1:32931|FINALIZED]]} size 0
   [junit4]   2> 3200573 WARN  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3200600 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3200600 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3200600 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3200608 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3200608 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3200609 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1961424476, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3200612 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@515d848f[collection1] main]
   [junit4]   2> 3200613 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3200613 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3200613 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3200615 INFO  (searcherExecutor-13406-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@515d848f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3200616 INFO  (coreLoadExecutor-13405-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570633868307857408
   [junit4]   2> 3200620 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3200620 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3200621 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55311/nst/fw/collection1/
   [junit4]   2> 3200621 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3200621 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:55311/nst/fw/collection1/ has no replicas
   [junit4]   2> 3200621 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3200623 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55311/nst/fw/collection1/ shard1
   [junit4]   2> 3200774 INFO  (coreZkRegister-13398-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw x:collection1 c:control_collection) [n:127.0.0.1:55311_nst%2Ffw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3200898 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3200899 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58917/solr ready
   [junit4]   2> 3200899 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3200988 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-1-001/cores/collection1
   [junit4]   2> 3200988 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-1-001
   [junit4]   2> 3200989 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3200992 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@39ea6b5{/nst/fw,null,AVAILABLE}
   [junit4]   2> 3200992 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2aa50014{HTTP/1.1,[http/1.1]}{127.0.0.1:38778}
   [junit4]   2> 3200992 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server Started @3203077ms
   [junit4]   2> 3200992 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:36966/hdfs__localhost_36966__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/nst/fw, hostPort=38778, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-1-001/cores}
   [junit4]   2> 3200994 ERROR (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3200994 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3200995 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3200995 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3200995 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T11:53:01.062Z
   [junit4]   2> 3200997 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3200997 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-1-001/solr.xml
   [junit4]   2> 3201005 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3201006 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58917/solr
   [junit4]   2> 3201021 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3201023 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3201025 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38778_nst%2Ffw
   [junit4]   2> 3201026 INFO  (zkCallback-31827-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3201030 INFO  (zkCallback-31831-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3201033 INFO  (zkCallback-31836-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3201115 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-1-001/cores
   [junit4]   2> 3201115 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3201116 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3201116 INFO  (OverseerStateUpdate-98164616659206148-127.0.0.1:55311_nst%2Ffw-n_0000000000) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3202127 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3202147 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3202265 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3202284 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3202285 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:36966/solr_hdfs_home
   [junit4]   2> 3202285 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3202285 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3202285 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:36966/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 3202285 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78d9b800
   [junit4]   2> 3202285 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 3202292 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3202292 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3202292 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3202297 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3202298 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 3202312 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 3202317 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3202317 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3202317 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3202322 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3202322 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3202341 INFO  (IPC Server handler 2 on 36966) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32931 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-43345fcd-5e09-4aa7-925b-a6e9805c0eaf:NORMAL:127.0.0.1:32931|RBW], ReplicaUC[[DISK]DS-96f0454d-71a1-441a-bb20-648eff417ff3:NORMAL:127.0.0.1:33736|RBW]]} size 0
   [junit4]   2> 3202341 INFO  (IPC Server handler 1 on 36966) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33736 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-43345fcd-5e09-4aa7-925b-a6e9805c0eaf:NORMAL:127.0.0.1:32931|RBW], ReplicaUC[[DISK]DS-35bfeff3-61da-4bcf-b500-98d455d027e8:NORMAL:127.0.0.1:33736|FINALIZED]]} size 0
   [junit4]   2> 3202346 WARN  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3203295 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3203295 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3203295 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3203303 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3203303 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3203305 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1961424476, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3203308 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6eec6405[collection1] main]
   [junit4]   2> 3203309 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3203309 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3203310 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3203311 INFO  (searcherExecutor-13417-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6eec6405[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3203312 INFO  (coreLoadExecutor-13416-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570633871134818304
   [junit4]   2> 3203317 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3203317 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3203317 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38778/nst/fw/collection1/
   [junit4]   2> 3203318 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3203319 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:38778/nst/fw/collection1/ has no replicas
   [junit4]   2> 3203319 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3203322 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38778/nst/fw/collection1/ shard1
   [junit4]   2> 3203474 INFO  (coreZkRegister-13411-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3203837 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-2-001/cores/collection1
   [junit4]   2> 3203838 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-2-001
   [junit4]   2> 3203838 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3203839 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5ffa6be9{/nst/fw,null,AVAILABLE}
   [junit4]   2> 3203840 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@67961839{HTTP/1.1,[http/1.1]}{127.0.0.1:52478}
   [junit4]   2> 3203840 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server Started @3205924ms
   [junit4]   2> 3203840 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:36966/hdfs__localhost_36966__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/nst/fw, hostPort=52478, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-2-001/cores}
   [junit4]   2> 3203840 ERROR (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3203841 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3203842 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3203842 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3203842 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T11:53:03.909Z
   [junit4]   2> 3203844 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3203844 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-2-001/solr.xml
   [junit4]   2> 3203850 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3203850 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58917/solr
   [junit4]   2> 3203857 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3203858 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3203860 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52478_nst%2Ffw
   [junit4]   2> 3203861 INFO  (zkCallback-31836-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3203861 INFO  (zkCallback-31831-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3203861 INFO  (zkCallback-31827-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3203866 INFO  (zkCallback-31842-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3203938 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-2-001/cores
   [junit4]   2> 3203938 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3203938 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3203940 INFO  (OverseerStateUpdate-98164616659206148-127.0.0.1:55311_nst%2Ffw-n_0000000000) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3204950 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3204962 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3205063 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3205073 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3205074 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:36966/solr_hdfs_home
   [junit4]   2> 3205074 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3205074 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3205074 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:36966/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 3205074 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78d9b800
   [junit4]   2> 3205075 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 3205084 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3205084 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3205084 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3205091 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3205092 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 3205109 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 3205116 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3205116 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3205116 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3205123 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3205123 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3205130 INFO  (IPC Server handler 1 on 36966) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32931 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-96f0454d-71a1-441a-bb20-648eff417ff3:NORMAL:127.0.0.1:33736|RBW], ReplicaUC[[DISK]DS-5284bd1e-8b3d-4e0a-9186-4be1c3da4cd2:NORMAL:127.0.0.1:32931|FINALIZED]]} size 0
   [junit4]   2> 3205130 INFO  (IPC Server handler 4 on 36966) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33736 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-96f0454d-71a1-441a-bb20-648eff417ff3:NORMAL:127.0.0.1:33736|RBW], ReplicaUC[[DISK]DS-5284bd1e-8b3d-4e0a-9186-4be1c3da4cd2:NORMAL:127.0.0.1:32931|FINALIZED]]} size 0
   [junit4]   2> 3205136 WARN  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3205171 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3205171 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3205171 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3205183 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3205183 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3205185 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1961424476, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3205188 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@668d9eeb[collection1] main]
   [junit4]   2> 3205189 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3205189 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3205189 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3205190 INFO  (searcherExecutor-13428-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@668d9eeb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3205192 INFO  (coreLoadExecutor-13427-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570633873106141184
   [junit4]   2> 3205196 INFO  (coreZkRegister-13422-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 c:collection1) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 3205196 INFO  (updateExecutor-31839-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 3205196 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 3205197 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 3205197 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 3205197 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 3205197 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:38778/nst/fw/collection1/] and I am [http://127.0.0.1:52478/nst/fw/collection1/]
   [junit4]   2> 3205199 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:38778/nst/fw]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:52478_nst%252Ffw&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 3205200 INFO  (qtp195928662-129720) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 3205200 INFO  (qtp195928662-129720) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 3205200 INFO  (qtp195928662-129720) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:52478_nst%2Ffw, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:52478/nst/fw","node_name":"127.0.0.1:52478_nst%2Ffw","state":"down"}
   [junit4]   2> 3205526 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-3-001/cores/collection1
   [junit4]   2> 3205526 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-3-001
   [junit4]   2> 3205527 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3205528 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@47490617{/nst/fw,null,AVAILABLE}
   [junit4]   2> 3205528 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1cd8785c{HTTP/1.1,[http/1.1]}{127.0.0.1:54118}
   [junit4]   2> 3205529 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.e.j.s.Server Started @3207613ms
   [junit4]   2> 3205529 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:36966/hdfs__localhost_36966__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001_tempDir-002_jetty3, solrconfig=solrconfig.xml, hostContext=/nst/fw, hostPort=54118, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-3-001/cores}
   [junit4]   2> 3205530 ERROR (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3205530 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3205530 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3205530 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3205530 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-19T11:53:05.597Z
   [junit4]   2> 3205532 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3205533 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-3-001/solr.xml
   [junit4]   2> 3205538 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3205538 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58917/solr
   [junit4]   2> 3205546 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:54118_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3205547 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:54118_nst%2Ffw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3205549 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:54118_nst%2Ffw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54118_nst%2Ffw
   [junit4]   2> 3205550 INFO  (zkCallback-31827-thread-1-processing-n:127.0.0.1:55311_nst%2Ffw) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3205550 INFO  (zkCallback-31836-thread-1-processing-n:127.0.0.1:38778_nst%2Ffw) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3205550 INFO  (zkCallback-31831-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3205550 INFO  (zkCallback-31842-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw) [n:127.0.0.1:52478_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3205551 INFO  (zkCallback-31849-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3205655 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:54118_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-3-001/cores
   [junit4]   2> 3205655 INFO  (TEST-StressHdfsTest.test-seed#[2C5F898E344E9D7D]) [n:127.0.0.1:54118_nst%2Ffw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3205656 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3205656 INFO  (OverseerStateUpdate-98164616659206148-127.0.0.1:55311_nst%2Ffw-n_0000000000) [n:127.0.0.1:55311_nst%2Ffw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3206201 INFO  (qtp195928662-129720) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:52478_nst%2Ffw, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","dataDir":"hdfs://localhost:36966/solr_hdfs_home/collection1/core_node2/data/","base_url":"http://127.0.0.1:52478/nst/fw","node_name":"127.0.0.1:52478_nst%2Ffw","state":"recovering","ulogDir":"hdfs://localhost:36966/solr_hdfs_home/collection1/core_node2/data/tlog"}
   [junit4]   2> 3206201 INFO  (qtp195928662-129720) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 3206201 INFO  (qtp195928662-129720) [n:127.0.0.1:38778_nst%2Ffw    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:52478_nst%252Ffw&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
   [junit4]   2> 3206676 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3206687 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3206701 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:38778/nst/fw/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 3206701 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:52478/nst/fw START replicas=[http://127.0.0.1:38778/nst/fw/collection1/] nUpdates=100
   [junit4]   2> 3206704 INFO  (qtp195928662-129722) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3206704 INFO  (qtp195928662-129722) [n:127.0.0.1:38778_nst%2Ffw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/nst/fw path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 3206705 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 3206706 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 3206706 INFO  (recoveryExecutor-31840-thread-1-processing-n:127.0.0.1:52478_nst%2Ffw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52478_nst%2Ffw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 3206787 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3206795 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3206795 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:36966/solr_hdfs_home
   [junit4]   2> 3206795 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3206796 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3206796 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001/shard-3-001/cores/collection1], dataDir=[hdfs://localhost:36966/solr_hdfs_home/collection1/core_node3/data/]
   [junit4]   2> 3206796 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78d9b800
   [junit4]   2> 3206796 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node3/data/snapshot_metadata
   [junit4]   2> 3206802 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3206802 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3206802 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3206809 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3206809 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node3/data
   [junit4]   2> 3206821 INFO  (coreLoadExecutor-13438-thread-1-processing-n:127.0.0.1:54118_nst%2Ffw) [n:127.0.0.1:54118_nst%2Ffw c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36966/solr_hdfs_home/collection1/core_node3/data/index
   [junit4]   2> 3206826 INFO  (coreLoadExecutor-13438-thread-1-processing

[...truncated too long message...]

ricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:203)
   [junit4]   2> at org.apache.solr.cloud.hdfs.StressHdfsTest.teardownClass(StressHdfsTest.java:77)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 53 more
   [junit4]   2> 3384324 INFO  (SUITE-StressHdfsTest-seed#[2C5F898E344E9D7D]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_2C5F898E344E9D7D-001
   [junit4]   2> Jun 19, 2017 11:56:04 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {range_facet_l_dv=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=PostingsFormat(name=LuceneVarGapFixedInterval), intDefault=PostingsFormat(name=Asserting), id_i1=PostingsFormat(name=Asserting), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_i_dv=PostingsFormat(name=LuceneVarGapFixedInterval), text=Lucene50(blocksize=128), range_facet_l=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=PostingsFormat(name=LuceneVarGapFixedInterval), txt_t=Lucene50(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Lucene54), multiDefault=DocValuesFormat(name=Memory), intDefault=DocValuesFormat(name=Lucene54), id_i1=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Asserting), text=DocValuesFormat(name=Lucene54), intDvoDefault=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Memory), range_facet_l=DocValuesFormat(name=Memory), txt_t=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=186, maxMBSortInHeap=6.205771100611032, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=ar, timezone=Asia/Aqtau
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=268578224,total=535298048
   [junit4]   2> NOTE: All tests run in this JVM: [TestSortByMinMaxFunction, TestFieldCacheSanityChecker, CurrencyFieldOpenExchangeTest, PeerSyncTest, SpatialRPTFieldTypeTest, ScriptEngineTest, TestReloadAndDeleteDocs, TestDownShardTolerantSearch, TestHighFrequencyDictionaryFactory, TestFunctionQuery, TestValueSourceCache, TestReplicaProperties, LargeFieldTest, TestReRankQParserPlugin, UniqFieldsUpdateProcessorFactoryTest, ClusterStateTest, SignatureUpdateProcessorFactoryTest, AlternateDirectoryTest, TestScoreJoinQPScore, TestFieldSortValues, DistributedFacetPivotSmallAdvancedTest, PathHierarchyTokenizerFactoryTest, ShufflingReplicaListTransformerTest, TestOmitPositions, TestBadConfig, CustomCollectionTest, TestPushWriter, TestSolrCoreProperties, TestGeoJSONResponseWriter, DistributedQueryComponentCustomSortTest, OverseerStatusTest, SortSpecParsingTest, JsonValidatorTest, LeaderElectionIntegrationTest, TestJoin, FullHLLTest, TestCodecSupport, ChangedSchemaMergeTest, TestSchemaResource, TestReplicationHandler, TestAuthenticationFramework, TestCrossCoreJoin, BlockJoinFacetRandomTest, PolyFieldTest, AliasIntegrationTest, TestSolrDeletionPolicy2, TestSizeLimitedDistributedMap, SecurityConfHandlerTest, TestSha256AuthenticationProvider, TestQuerySenderNoQuery, TestSSLRandomization, TestJsonFacets, CoreAdminCreateDiscoverTest, TestBlendedInfixSuggestions, TestJmxIntegration, BasicZkTest, TestSolr4Spatial2, TestDocBasedVersionConstraints, DistributedFacetExistsSmallTest, TestHashQParserPlugin, SuggestComponentTest, TestInPlaceUpdatesDistrib, SyncSliceTest, RegexBoostProcessorTest, StandardRequestHandlerTest, TestIBSimilarityFactory, TestHttpShardHandlerFactory, TestJettySolrRunner, TestFoldingMultitermQuery, ReplicationFactorTest, TestRandomFaceting, TestSolrCloudWithHadoopAuthPlugin, HighlighterConfigTest, ReturnFieldsTest, TestXmlQParser, ExitableDirectoryReaderTest, TestStressLiveNodes, TestQueryWrapperFilter, QueryEqualityTest, TestMacros, UUIDFieldTest, TestBinaryResponseWriter, TestLazyCores, ConfigSetsAPITest, RecoveryAfterSoftCommitTest, TestClusterStateMutator, TestSolrCloudSnapshots, TestSerializedLuceneMatchVersion, SolrCLIZkUtilsTest, BasicAuthIntegrationTest, TestCloudRecovery, TestSolrConfigHandler, TestCloudNestedDocsSort, TestMissingGroups, SharedFSAutoReplicaFailoverTest, DocValuesNotIndexedTest, TestNumericTerms32, TestCoreAdminApis, TestInfoStreamLogging, OverseerCollectionConfigSetProcessorTest, DistributedQueryElevationComponentTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSolrConfigHandlerConcurrent, DistribJoinFromCollectionTest, AnalyticsMergeStrategyTest, ConfigureRecoveryStrategyTest, TestClassNameShortening, TestFaceting, BaseCdcrDistributedZkTest, NumericFieldsTest, TestSolr4Spatial, HdfsBasicDistributedZk2Test, ClassificationUpdateProcessorTest, TestSchemalessBufferedUpdates, BasicDistributedZkTest, RecoveryZkTest, ClusterStateUpdateTest, ZkSolrClientTest, ShardRoutingCustomTest, TestHashPartitioner, TermVectorComponentDistributedTest, TestReload, TestRangeQuery, TestMergePolicyConfig, MultiTermTest, TestDocSet, TestSearchPerf, MinimalSchemaTest, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationHandlerTest, ConnectionManagerTest, DeleteNodeTest, DeleteStatusTest, DistribDocExpirationUpdateProcessorTest, MultiThreadedOCPTest, TestCloudJSONFacetJoinDomain, TestConfigSetsAPI, TestLeaderElectionZkExpiry, TestLocalFSCloudBackupRestore, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestSolrCloudWithKerberosAlt, TestStressInPlaceUpdates, TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, HdfsChaosMonkeySafeLeaderTest, HdfsUnloadDistributedZkTest, StressHdfsTest]
   [junit4] Completed [596/715 (1!)] on J0 in 196.67s, 1 test, 1 error <<< FAILURES!

[...truncated 43820 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 376 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/376/

3 tests failed.
FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:35625/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:58748/solr/test_col_shard1_replica2: Server Error    request: http://127.0.0.1:58748/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A35625%2Fsolr%2Ftest_col_shard2_replica2%2F&wt=javabin&version=2 Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:48366/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@1d368b0b

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:35625/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:58748/solr/test_col_shard1_replica2: Server Error



request: http://127.0.0.1:58748/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A35625%2Fsolr%2Ftest_col_shard2_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:48366/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@1d368b0b
        at __randomizedtesting.SeedInfo.seed([E062A65070CA1B5D:D676C416FA97214C]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:35625/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:58748/solr/test_col_shard1_replica2: Server Error



request: http://127.0.0.1:58748/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A35625%2Fsolr%2Ftest_col_shard2_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:48366/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@1d368b0b
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35625/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:58748/solr/test_col_shard1_replica2: Server Error



request: http://127.0.0.1:58748/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A35625%2Fsolr%2Ftest_col_shard2_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:48366/solr/test_col_shard1_replica1/ update: org.apache.solr.client.solrj.request.UpdateRequest@1d368b0b
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:594)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
        ... 6 more


FAILED:  org.apache.solr.schema.TestUseDocValuesAsStored.testMultipleSearchResults

Error Message:
mismatch: 'myid1'!='myid' @ response/docs/[0]/id

Stack Trace:
java.lang.RuntimeException: mismatch: 'myid1'!='myid' @ response/docs/[0]/id
        at __randomizedtesting.SeedInfo.seed([E062A65070CA1B5D:D248A1CE88343F84]:0)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:983)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:930)
        at org.apache.solr.schema.TestUseDocValuesAsStored.testMultipleSearchResults(TestUseDocValuesAsStored.java:243)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:37665

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:37665
        at __randomizedtesting.SeedInfo.seed([E062A65070CA1B5D:6836998ADE3676A5]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:605)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:477)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:407)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:497)
        ... 51 more




Build Log:
[...truncated 12275 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/init-core-data-001
   [junit4]   2> 1076436 WARN  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1076436 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 1076438 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1076439 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1076478 WARN  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1076485 WARN  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1076487 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1076503 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_40990_hdfs____vyotry/webapp
   [junit4]   2> 1076854 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40990
   [junit4]   2> 1076921 WARN  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1076922 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1076934 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_52234_datanode____o07yv1/webapp
   [junit4]   2> 1077322 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52234
   [junit4]   2> 1077380 WARN  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1077381 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1077395 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_42670_datanode____gad4u0/webapp
   [junit4]   2> 1077462 INFO  (IPC Server handler 0 on 33473) [    ] BlockStateChange BLOCK* processReport: from storage DS-20bcacd2-6bcd-4e74-b893-bd133310de42 node DatanodeRegistration(127.0.0.1:55837, datanodeUuid=8569eda5-d157-4a8c-9692-1ced156ad927, infoPort=59749, infoSecurePort=0, ipcPort=49687, storageInfo=lv=-56;cid=testClusterID;nsid=1959368272;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1077462 INFO  (IPC Server handler 0 on 33473) [    ] BlockStateChange BLOCK* processReport: from storage DS-b870a836-b4b6-4094-9782-7464f86b3f36 node DatanodeRegistration(127.0.0.1:55837, datanodeUuid=8569eda5-d157-4a8c-9692-1ced156ad927, infoPort=59749, infoSecurePort=0, ipcPort=49687, storageInfo=lv=-56;cid=testClusterID;nsid=1959368272;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1077798 INFO  (SUITE-StressHdfsTest-seed#[E062A65070CA1B5D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42670
   [junit4]   2> 1077876 INFO  (IPC Server handler 8 on 33473) [    ] BlockStateChange BLOCK* processReport: from storage DS-601bfb62-3d23-4b80-a6d7-6dfcab03a7a2 node DatanodeRegistration(127.0.0.1:60886, datanodeUuid=e9d2c234-876d-4036-a08e-b1de7340c147, infoPort=39843, infoSecurePort=0, ipcPort=55231, storageInfo=lv=-56;cid=testClusterID;nsid=1959368272;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1077876 INFO  (IPC Server handler 8 on 33473) [    ] BlockStateChange BLOCK* processReport: from storage DS-ebb53e23-785c-48d9-a69b-226bf058ae36 node DatanodeRegistration(127.0.0.1:60886, datanodeUuid=e9d2c234-876d-4036-a08e-b1de7340c147, infoPort=39843, infoSecurePort=0, ipcPort=55231, storageInfo=lv=-56;cid=testClusterID;nsid=1959368272;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1077965 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1077965 INFO  (Thread-3241) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1077965 INFO  (Thread-3241) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1077968 ERROR (Thread-3241) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1078065 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ZkTestServer start zk server on port:45421
   [junit4]   2> 1078077 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1078079 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1078081 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1078083 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1078084 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1078086 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1078087 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1078089 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1078090 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1078092 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1078094 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1078208 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/control-001/cores/collection1
   [junit4]   2> 1078210 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1078212 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6325fe25{/,null,AVAILABLE}
   [junit4]   2> 1078212 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5a2fa58b{HTTP/1.1,[http/1.1]}{127.0.0.1:37665}
   [junit4]   2> 1078213 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server Started @1080931ms
   [junit4]   2> 1078213 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33473/hdfs__localhost_33473__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001_tempDir-002_control_data, hostContext=/, hostPort=37665, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/control-001/cores}
   [junit4]   2> 1078213 ERROR (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1078213 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 1078213 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1078213 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1078213 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-20T12:46:52.112Z
   [junit4]   2> 1078216 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1078216 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/control-001/solr.xml
   [junit4]   2> 1078223 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1078224 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45421/solr
   [junit4]   2> 1078252 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37665_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1078253 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37665_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37665_
   [junit4]   2> 1078256 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37665_    ] o.a.s.c.Overseer Overseer (id=98170490838450180-127.0.0.1:37665_-n_0000000000) starting
   [junit4]   2> 1078267 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37665_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37665_
   [junit4]   2> 1078269 INFO  (zkCallback-1941-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1078356 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37665_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/control-001/cores
   [junit4]   2> 1078357 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37665_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1078358 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1078360 INFO  (OverseerStateUpdate-98170490838450180-127.0.0.1:37665_-n_0000000000) [n:127.0.0.1:37665_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1079375 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 1079391 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1079518 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1079531 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1079531 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33473/solr_hdfs_home
   [junit4]   2> 1079532 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1079532 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1079532 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/control-001/cores/collection1], dataDir=[hdfs://localhost:33473/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1079532 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a9935b3
   [junit4]   2> 1079532 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 1079540 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1079540 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1079540 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1079548 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1079549 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1079566 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1079582 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1079582 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1079582 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1079588 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1079589 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=648424779, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1079606 INFO  (IPC Server handler 0 on 33473) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55837 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-601bfb62-3d23-4b80-a6d7-6dfcab03a7a2:NORMAL:127.0.0.1:60886|RBW], ReplicaUC[[DISK]DS-20bcacd2-6bcd-4e74-b893-bd133310de42:NORMAL:127.0.0.1:55837|RBW]]} size 0
   [junit4]   2> 1079606 INFO  (IPC Server handler 5 on 33473) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60886 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-20bcacd2-6bcd-4e74-b893-bd133310de42:NORMAL:127.0.0.1:55837|RBW], ReplicaUC[[DISK]DS-ebb53e23-785c-48d9-a69b-226bf058ae36:NORMAL:127.0.0.1:60886|FINALIZED]]} size 0
   [junit4]   2> 1079616 WARN  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1079660 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1079660 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1079660 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1079671 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1079672 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1079673 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19065451385654564]
   [junit4]   2> 1079681 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@143e52bb[collection1] main]
   [junit4]   2> 1079682 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1079683 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1079683 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1079684 INFO  (searcherExecutor-3976-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@143e52bb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1079687 INFO  (coreLoadExecutor-3975-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570727855218753536
   [junit4]   2> 1079697 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1079698 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1079698 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37665/collection1/
   [junit4]   2> 1079698 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1079698 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:37665/collection1/ has no replicas
   [junit4]   2> 1079698 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1079702 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37665/collection1/ shard1
   [junit4]   2> 1079853 INFO  (coreZkRegister-3968-thread-1-processing-n:127.0.0.1:37665_ x:collection1 c:control_collection) [n:127.0.0.1:37665_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1079871 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1079872 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45421/solr ready
   [junit4]   2> 1079872 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1079872 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1080004 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-1-001/cores/collection1
   [junit4]   2> 1080005 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-1-001
   [junit4]   2> 1080005 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1080009 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c530235{/,null,AVAILABLE}
   [junit4]   2> 1080009 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@53dfdf5{HTTP/1.1,[http/1.1]}{127.0.0.1:55465}
   [junit4]   2> 1080009 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server Started @1082727ms
   [junit4]   2> 1080009 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33473/hdfs__localhost_33473__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=55465, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-1-001/cores}
   [junit4]   2> 1080009 ERROR (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1080009 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 1080010 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1080010 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1080010 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-20T12:46:53.909Z
   [junit4]   2> 1080013 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1080013 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-1-001/solr.xml
   [junit4]   2> 1080020 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1080021 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45421/solr
   [junit4]   2> 1080036 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1080039 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:55465_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1080041 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:55465_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55465_
   [junit4]   2> 1080043 INFO  (zkCallback-1945-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1080043 INFO  (zkCallback-1941-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1080045 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1080100 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:55465_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-1-001/cores
   [junit4]   2> 1080100 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:55465_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1080102 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1080103 INFO  (OverseerStateUpdate-98170490838450180-127.0.0.1:37665_-n_0000000000) [n:127.0.0.1:37665_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1080205 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1081116 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 1081145 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1081273 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1081286 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 1081287 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33473/solr_hdfs_home
   [junit4]   2> 1081287 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1081287 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1081287 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:33473/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 1081287 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a9935b3
   [junit4]   2> 1081287 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 1081295 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1081295 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1081295 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1081301 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1081302 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 1081319 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 1081325 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1081325 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1081325 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1081331 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1081332 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=648424779, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1081349 INFO  (IPC Server handler 9 on 33473) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55837 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-601bfb62-3d23-4b80-a6d7-6dfcab03a7a2:NORMAL:127.0.0.1:60886|RBW], ReplicaUC[[DISK]DS-b870a836-b4b6-4094-9782-7464f86b3f36:NORMAL:127.0.0.1:55837|RBW]]} size 0
   [junit4]   2> 1081351 INFO  (IPC Server handler 2 on 33473) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60886 is added to blk_1073741826_1002 size 71
   [junit4]   2> 1081360 WARN  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1081411 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1081411 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1081411 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1081422 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1081422 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1081424 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19065451385654564]
   [junit4]   2> 1081430 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6579e842[collection1] main]
   [junit4]   2> 1081431 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1081431 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1081431 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1081433 INFO  (searcherExecutor-3987-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6579e842[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1081435 INFO  (coreLoadExecutor-3986-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570727857051664384
   [junit4]   2> 1081442 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1081442 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1081442 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55465/collection1/
   [junit4]   2> 1081442 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1081442 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:55465/collection1/ has no replicas
   [junit4]   2> 1081443 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1081446 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55465/collection1/ shard1
   [junit4]   2> 1081549 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1081598 INFO  (coreZkRegister-3981-thread-1-processing-n:127.0.0.1:55465_ x:collection1 c:collection1) [n:127.0.0.1:55465_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1081703 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1081724 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-2-001/cores/collection1
   [junit4]   2> 1081724 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-2-001
   [junit4]   2> 1081725 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1081727 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@557f8bd8{/,null,AVAILABLE}
   [junit4]   2> 1081727 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@29d6f3e9{HTTP/1.1,[http/1.1]}{127.0.0.1:56506}
   [junit4]   2> 1081727 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server Started @1084445ms
   [junit4]   2> 1081727 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33473/hdfs__localhost_33473__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=56506, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-2-001/cores}
   [junit4]   2> 1081728 ERROR (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1081728 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 1081728 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1081728 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1081728 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-20T12:46:55.627Z
   [junit4]   2> 1081732 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1081732 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-2-001/solr.xml
   [junit4]   2> 1081746 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1081748 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45421/solr
   [junit4]   2> 1081762 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:56506_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1081764 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:56506_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1081767 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:56506_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56506_
   [junit4]   2> 1081768 INFO  (zkCallback-1941-thread-1-processing-n:127.0.0.1:37665_) [n:127.0.0.1:37665_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1081768 INFO  (zkCallback-1945-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1081768 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1081773 INFO  (zkCallback-1957-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1081875 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:56506_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-2-001/cores
   [junit4]   2> 1081876 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:56506_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1081877 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1081878 INFO  (OverseerStateUpdate-98170490838450180-127.0.0.1:37665_-n_0000000000) [n:127.0.0.1:37665_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1081986 INFO  (zkCallback-1957-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1081986 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1082898 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 1082919 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1083048 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1083062 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 1083062 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33473/solr_hdfs_home
   [junit4]   2> 1083062 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1083062 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1083062 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:33473/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 1083063 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a9935b3
   [junit4]   2> 1083063 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 1083071 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1083071 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1083071 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1083078 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1083079 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 1083096 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33473/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 1083102 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1083102 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1083102 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1083108 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1083109 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=648424779, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1083120 INFO  (IPC Server handler 8 on 33473) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60886 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b870a836-b4b6-4094-9782-7464f86b3f36:NORMAL:127.0.0.1:55837|RBW], ReplicaUC[[DISK]DS-ebb53e23-785c-48d9-a69b-226bf058ae36:NORMAL:127.0.0.1:60886|RBW]]} size 0
   [junit4]   2> 1083121 INFO  (IPC Server handler 9 on 33473) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55837 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ebb53e23-785c-48d9-a69b-226bf058ae36:NORMAL:127.0.0.1:60886|RBW], ReplicaUC[[DISK]DS-20bcacd2-6bcd-4e74-b893-bd133310de42:NORMAL:127.0.0.1:55837|FINALIZED]]} size 0
   [junit4]   2> 1083130 WARN  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1083177 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1083177 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1083177 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1083189 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1083189 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1083191 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19065451385654564]
   [junit4]   2> 1083199 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3acd5958[collection1] main]
   [junit4]   2> 1083201 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1083201 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1083201 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1083203 INFO  (searcherExecutor-3998-thread-1-processing-n:127.0.0.1:56506_ x:collection1 c:collection1) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3acd5958[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1083215 INFO  (coreLoadExecutor-3997-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570727858918129664
   [junit4]   2> 1083220 INFO  (coreZkRegister-3992-thread-1-processing-n:127.0.0.1:56506_ x:collection1 c:collection1) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 1083225 INFO  (updateExecutor-1954-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1083225 INFO  (recoveryExecutor-1955-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 1083226 INFO  (recoveryExecutor-1955-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 1083226 INFO  (recoveryExecutor-1955-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 1083226 INFO  (recoveryExecutor-1955-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1083226 INFO  (recoveryExecutor-1955-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:55465/collection1/] and I am [http://127.0.0.1:56506/collection1/]
   [junit4]   2> 1083236 INFO  (recoveryExecutor-1955-thread-1-processing-n:127.0.0.1:56506_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56506_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:55465]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:56506_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1083237 INFO  (qtp824049514-10666) [n:127.0.0.1:55465_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 1083238 INFO  (qtp824049514-10666) [n:127.0.0.1:55465_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 1083238 INFO  (qtp824049514-10666) [n:127.0.0.1:55465_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:56506_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:56506","node_name":"127.0.0.1:56506_","state":"down"}
   [junit4]   2> 1083332 INFO  (zkCallback-1951-thread-1-processing-n:127.0.0.1:55465_) [n:127.0.0.1:55465_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1083332 INFO  (zkCallback-1957-thread-1-processing-n:127.0.0.1:56506_) [n:127.0.0.1:56506_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1083519 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-3-001/cores/collection1
   [junit4]   2> 1083520 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-3-001
   [junit4]   2> 1083521 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1083546 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@65cea532{/,null,AVAILABLE}
   [junit4]   2> 1083546 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c1cdca6{HTTP/1.1,[http/1.1]}{127.0.0.1:37235}
   [junit4]   2> 1083546 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.e.j.s.Server Started @1086265ms
   [junit4]   2> 1083547 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33473/hdfs__localhost_33473__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001_tempDir-002_jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=37235, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-3-001/cores}
   [junit4]   2> 1083547 ERROR (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1083547 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 1083547 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1083547 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1083547 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-20T12:46:57.446Z
   [junit4]   2> 1083552 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1083552 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_E062A65070CA1B5D-001/shard-3-001/solr.xml
   [junit4]   2> 1083560 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1083560 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45421/solr
   [junit4]   2> 1083571 INFO  (TEST-StressHdfsTest.test-seed#[E062A65070CA1B5D]) [n:127.0.0.1:37235_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (

[...truncated too long message...]

ll params={q=*:*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4176626 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4176627 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=test_nonstored_dv_str&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4176627 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*,test_nonstored_dv_str&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4176627 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[xyz]} 0 0
   [junit4]   2> 4176628 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4176628 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@34da877b
   [junit4]   2> 4176644 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@21e05f09[collection1] main]
   [junit4]   2> 4176645 INFO  (searcherExecutor-8402-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@21e05f09[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_55(6.7.0):C1)))}
   [junit4]   2> 4176646 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4176646 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 18
   [junit4]   2> 4176646 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4176647 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4176647 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:*} 0 0
   [junit4]   2> 4176647 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 4176647 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1246902727
   [junit4]   2> 4176647 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 4176648 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 4176650 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 4176650 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4176650 INFO  (coreCloseExecutor-8406-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@28bac6ec
   [junit4]   2> 4176651 INFO  (coreCloseExecutor-8406-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 4176653 INFO  (TEST-TestUseDocValuesAsStored.testOnEmptyIndex-seed#[E062A65070CA1B5D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testOnEmptyIndex
   [junit4]   2> 4176654 INFO  (SUITE-TestUseDocValuesAsStored-seed#[E062A65070CA1B5D]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.schema.TestUseDocValuesAsStored_E062A65070CA1B5D-001
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=989, maxDocsPerChunk=8, blockSize=2), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=989, blockSize=2)), sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=ms, timezone=Pacific/Norfolk
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=214692256,total=534249472
   [junit4]   2> NOTE: All tests run in this JVM: [SyncSliceTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, OverseerCollectionConfigSetProcessorTest, TestAtomicUpdateErrorCases, AnalysisAfterCoreReloadTest, NodeMutatorTest, TestLazyCores, TestReRankQParserPlugin, TestDocumentBuilder, TestComponentsName, TestManagedResourceStorage, FacetPivotSmallTest, TestCustomStream, TestSolrCloudSnapshots, ClassificationUpdateProcessorTest, PathHierarchyTokenizerFactoryTest, JsonValidatorTest, BigEndianAscendingWordSerializerTest, CursorPagingTest, UUIDFieldTest, UniqFieldsUpdateProcessorFactoryTest, TestIBSimilarityFactory, RegexBoostProcessorTest, ReturnFieldsTest, TestCryptoKeys, FileBasedSpellCheckerTest, LargeFieldTest, TestSolr4Spatial, TestSizeLimitedDistributedMap, TestHighlightDedupGrouping, TestSolrConfigHandler, TestMacros, TestNumericTerms32, CollectionsAPIAsyncDistributedZkTest, TestValueSourceCache, SimpleMLTQParserTest, BaseCdcrDistributedZkTest, InfoHandlerTest, RequiredFieldsTest, BasicDistributedZkTest, ChangedSchemaMergeTest, TestSolrQueryResponse, QueryEqualityTest, UtilsToolTest, HdfsRecoverLeaseTest, TestJmxIntegration, TestBinaryResponseWriter, TestQueryWrapperFilter, ConfigureRecoveryStrategyTest, TestReplicationHandler, DocumentAnalysisRequestHandlerTest, TestJettySolrRunner, JavabinLoaderTest, TestAuthenticationFramework, TestSlowCompositeReaderWrapper, HdfsBasicDistributedZk2Test, BasicZkTest, TestBadConfig, TestFieldTypeCollectionResource, TestConfigSetProperties, TestLocalFSCloudBackupRestore, AnalyticsMergeStrategyTest, SortSpecParsingTest, SynonymTokenizerTest, BlobRepositoryCloudTest, TestStandardQParsers, TestDynamicFieldResource, SuggestComponentTest, SolrSlf4jReporterTest, ConfigSetsAPITest, SignatureUpdateProcessorFactoryTest, TestSearcherReuse, TestDistributedSearch, TestUniqueKeyFieldResource, TestHttpShardHandlerFactory, DataDrivenBlockJoinTest, AlternateDirectoryTest, TestExactSharedStatsCache, TestReplicationHandlerBackup, TestRandomFlRTGCloud, BlockJoinFacetRandomTest, SparseHLLTest, TestReplicaProperties, NotRequiredUniqueKeyTest, TestSolrConfigHandlerConcurrent, SSLMigrationTest, BasicAuthIntegrationTest, CdcrReplicationHandlerTest, ExternalFileFieldSortTest, HdfsSyncSliceTest, TestCloudJSONFacetJoinDomain, CustomCollectionTest, TestFieldSortValues, LeaderInitiatedRecoveryOnCommitTest, TestCrossCoreJoin, TestHighFrequencyDictionaryFactory, TestMissingGroups, TestSolrDeletionPolicy2, CoreAdminCreateDiscoverTest, TestRTimerTree, DeleteNodeTest, TestFieldCacheVsDocValues, TestSchemaResource, DistribJoinFromCollectionTest, OutputWriterTest, TestStressLucene, TestJoin, TestHalfAndHalfDocValues, TestCodecSupport, TestBackupRepositoryFactory, MBeansHandlerTest, TestBinaryResponseWriter, CollectionReloadTest, TestSerializedLuceneMatchVersion, TestSchemalessBufferedUpdates, SolrCoreTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestCloudRecovery, TestConfigSetsAPI, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, TestDownShardTolerantSearch, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, TestMiniSolrCloudCluster, TestMiniSolrCloudClusterSSL, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressCloudBlindAtomicUpdates, HdfsChaosMonkeyNothingIsSafeTest, HdfsRestartWhileUpdatingTest, TestClusterStateMutator, ZkStateWriterTest, RulesTest, CoreSorterTest, DirectoryFactoryTest, ExitableDirectoryReaderTest, QueryResultKeyTest, TestCorePropertiesReload, TestImplicitCoreProperties, TestInfoStreamLogging, TestInitParams, TestReloadAndDeleteDocs, TestShardHandlerFactory, TestSolrCoreSnapshots, TestConfigReload, TestSQLHandler, DistributedFacetPivotLongTailTest, ReplicaListTransformerTest, ShufflingReplicaListTransformerTest, TestExpandComponent, TestTrackingShardHandlerFactory, SolrJmxReporterTest, SmileWriterTest, SubstringBytesRefFilterTest, TestIntervalFaceting, TestPushWriter, TestRawResponseWriter, TestChildDocTransformer, TestManagedResource, TestRestManager, TestClassNameShortening, TestCopyFieldCollectionResource, TestFieldCollectionResource, TestFieldResource, TestSchemaNameResource, TestSchemaSimilarityResource, TestSchemaVersionResource, TestManagedStopFilterFactory, TestManagedSynonymGraphFilterFactory, DateRangeFieldTest, SpatialRPTFieldTypeTest, TestBulkSchemaConcurrent, TestUseDocValuesAsStored]
   [junit4] Completed [638/715 (3!)] on J0 in 13.58s, 6 tests, 1 error <<< FAILURES!

[...truncated 43678 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 377 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/377/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:50095/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:54890/solr/test_col_shard2_replica1: Server Error    request: http://127.0.0.1:54890/solr/test_col_shard2_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A50095%2Fsolr%2Ftest_col_shard5_replica1%2F&wt=javabin&version=2 Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42713/solr/test_col_shard2_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@21357254

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:50095/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:54890/solr/test_col_shard2_replica1: Server Error



request: http://127.0.0.1:54890/solr/test_col_shard2_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A50095%2Fsolr%2Ftest_col_shard5_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42713/solr/test_col_shard2_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@21357254
        at __randomizedtesting.SeedInfo.seed([3CDFB00F0E4D1BDC:ACBD249841021CD]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:50095/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:54890/solr/test_col_shard2_replica1: Server Error



request: http://127.0.0.1:54890/solr/test_col_shard2_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A50095%2Fsolr%2Ftest_col_shard5_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42713/solr/test_col_shard2_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@21357254
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50095/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:54890/solr/test_col_shard2_replica1: Server Error



request: http://127.0.0.1:54890/solr/test_col_shard2_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A50095%2Fsolr%2Ftest_col_shard5_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42713/solr/test_col_shard2_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@21357254
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:594)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
        ... 6 more




Build Log:
[...truncated 13201 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/init-core-data-001
   [junit4]   2> 3228942 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 3228943 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=SSL overhead seems to cause OutOfMemory when stress testing)
   [junit4]   2> 3228943 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 10 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001
   [junit4]   2> 3228943 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3228944 INFO  (Thread-61944) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3228944 INFO  (Thread-61944) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3228947 ERROR (Thread-61944) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3229044 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:33933
   [junit4]   2> 3229050 INFO  (jetty-launcher-27550-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229050 INFO  (jetty-launcher-27550-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229054 INFO  (jetty-launcher-27550-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229055 INFO  (jetty-launcher-27550-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e0d2059{/solr,null,AVAILABLE}
   [junit4]   2> 3229061 INFO  (jetty-launcher-27550-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229062 INFO  (jetty-launcher-27550-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1456537c{HTTP/1.1,[http/1.1]}{127.0.0.1:43223}
   [junit4]   2> 3229062 INFO  (jetty-launcher-27550-thread-2) [    ] o.e.j.s.Server Started @3232211ms
   [junit4]   2> 3229062 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43223}
   [junit4]   2> 3229062 ERROR (jetty-launcher-27550-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229063 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229063 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229063 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229063 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.335Z
   [junit4]   2> 3229064 INFO  (jetty-launcher-27550-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f636dae{/solr,null,AVAILABLE}
   [junit4]   2> 3229064 INFO  (jetty-launcher-27550-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3cb2b0c4{HTTP/1.1,[http/1.1]}{127.0.0.1:50095}
   [junit4]   2> 3229064 INFO  (jetty-launcher-27550-thread-3) [    ] o.e.j.s.Server Started @3232213ms
   [junit4]   2> 3229064 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50095}
   [junit4]   2> 3229065 ERROR (jetty-launcher-27550-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229065 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229065 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229065 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229065 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.337Z
   [junit4]   2> 3229076 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229077 INFO  (jetty-launcher-27550-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229077 INFO  (jetty-launcher-27550-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@135b21b{/solr,null,AVAILABLE}
   [junit4]   2> 3229079 INFO  (jetty-launcher-27550-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e7c28b0{/solr,null,AVAILABLE}
   [junit4]   2> 3229079 INFO  (jetty-launcher-27550-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1ef7c7b2{HTTP/1.1,[http/1.1]}{127.0.0.1:54890}
   [junit4]   2> 3229079 INFO  (jetty-launcher-27550-thread-4) [    ] o.e.j.s.Server Started @3232228ms
   [junit4]   2> 3229079 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54890}
   [junit4]   2> 3229079 ERROR (jetty-launcher-27550-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229080 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229080 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229080 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229080 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.352Z
   [junit4]   2> 3229084 INFO  (jetty-launcher-27550-thread-7) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229084 INFO  (jetty-launcher-27550-thread-8) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229084 INFO  (jetty-launcher-27550-thread-6) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229085 INFO  (jetty-launcher-27550-thread-9) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229085 INFO  (jetty-launcher-27550-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6aab504f{HTTP/1.1,[http/1.1]}{127.0.0.1:42835}
   [junit4]   2> 3229085 INFO  (jetty-launcher-27550-thread-1) [    ] o.e.j.s.Server Started @3232234ms
   [junit4]   2> 3229085 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42835}
   [junit4]   2> 3229091 ERROR (jetty-launcher-27550-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229091 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229091 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229091 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229091 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.363Z
   [junit4]   2> 3229090 INFO  (jetty-launcher-27550-thread-10) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3229089 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229092 INFO  (jetty-launcher-27550-thread-8) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@54a183e8{/solr,null,AVAILABLE}
   [junit4]   2> 3229092 INFO  (jetty-launcher-27550-thread-8) [    ] o.e.j.s.AbstractConnector Started ServerConnector@774c5354{HTTP/1.1,[http/1.1]}{127.0.0.1:46589}
   [junit4]   2> 3229092 INFO  (jetty-launcher-27550-thread-8) [    ] o.e.j.s.Server Started @3232242ms
   [junit4]   2> 3229093 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229093 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46589}
   [junit4]   2> 3229093 ERROR (jetty-launcher-27550-thread-8) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229098 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229098 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229098 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229098 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.370Z
   [junit4]   2> 3229103 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229104 INFO  (jetty-launcher-27550-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229104 INFO  (jetty-launcher-27550-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7cac8bd3{/solr,null,AVAILABLE}
   [junit4]   2> 3229114 INFO  (jetty-launcher-27550-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@12c8e50c{HTTP/1.1,[http/1.1]}{127.0.0.1:40978}
   [junit4]   2> 3229114 INFO  (jetty-launcher-27550-thread-6) [    ] o.e.j.s.Server Started @3232263ms
   [junit4]   2> 3229114 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40978}
   [junit4]   2> 3229115 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229115 INFO  (jetty-launcher-27550-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29efadfa{/solr,null,AVAILABLE}
   [junit4]   2> 3229116 INFO  (jetty-launcher-27550-thread-7) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6b9f811{HTTP/1.1,[http/1.1]}{127.0.0.1:45361}
   [junit4]   2> 3229116 INFO  (jetty-launcher-27550-thread-7) [    ] o.e.j.s.Server Started @3232265ms
   [junit4]   2> 3229116 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45361}
   [junit4]   2> 3229116 INFO  (jetty-launcher-27550-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229117 INFO  (jetty-launcher-27550-thread-10) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7607ee4{/solr,null,AVAILABLE}
   [junit4]   2> 3229117 INFO  (jetty-launcher-27550-thread-9) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@34f219cb{/solr,null,AVAILABLE}
   [junit4]   2> 3229118 INFO  (jetty-launcher-27550-thread-10) [    ] o.e.j.s.AbstractConnector Started ServerConnector@60aba417{HTTP/1.1,[http/1.1]}{127.0.0.1:33640}
   [junit4]   2> 3229118 INFO  (jetty-launcher-27550-thread-10) [    ] o.e.j.s.Server Started @3232267ms
   [junit4]   2> 3229118 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33640}
   [junit4]   2> 3229118 ERROR (jetty-launcher-27550-thread-6) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229119 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229119 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229119 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229119 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.391Z
   [junit4]   2> 3229120 ERROR (jetty-launcher-27550-thread-10) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229121 INFO  (jetty-launcher-27550-thread-9) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2ebda9cb{HTTP/1.1,[http/1.1]}{127.0.0.1:42713}
   [junit4]   2> 3229123 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229124 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229124 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229124 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.396Z
   [junit4]   2> 3229121 INFO  (jetty-launcher-27550-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@494d1c67{/solr,null,AVAILABLE}
   [junit4]   2> 3229123 ERROR (jetty-launcher-27550-thread-7) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229123 INFO  (jetty-launcher-27550-thread-9) [    ] o.e.j.s.Server Started @3232272ms
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42713}
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.398Z
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229126 ERROR (jetty-launcher-27550-thread-9) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229126 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.398Z
   [junit4]   2> 3229130 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229130 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229133 INFO  (jetty-launcher-27550-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229134 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229142 INFO  (jetty-launcher-27550-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@71bc9403{HTTP/1.1,[http/1.1]}{127.0.0.1:49436}
   [junit4]   2> 3229143 INFO  (jetty-launcher-27550-thread-5) [    ] o.e.j.s.Server Started @3232292ms
   [junit4]   2> 3229143 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229143 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ccaf5a5110007, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3229143 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49436}
   [junit4]   2> 3229145 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229146 ERROR (jetty-launcher-27550-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3229152 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.7.0
   [junit4]   2> 3229152 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3229152 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3229152 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-21T14:01:55.424Z
   [junit4]   2> 3229155 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229156 INFO  (jetty-launcher-27550-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229159 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229160 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229161 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229162 INFO  (jetty-launcher-27550-thread-9) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229162 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229162 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229164 INFO  (jetty-launcher-27550-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229165 INFO  (jetty-launcher-27550-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229168 INFO  (jetty-launcher-27550-thread-8) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229182 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3229186 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229188 INFO  (jetty-launcher-27550-thread-10) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229196 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3229199 INFO  (jetty-launcher-27550-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33933/solr
   [junit4]   2> 3229217 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ccaf5a511001c, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3229220 INFO  (jetty-launcher-27550-thread-4) [n:127.0.0.1:54890_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229220 INFO  (jetty-launcher-27550-thread-8) [n:127.0.0.1:46589_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229221 INFO  (jetty-launcher-27550-thread-2) [n:127.0.0.1:43223_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229221 INFO  (jetty-launcher-27550-thread-1) [n:127.0.0.1:42835_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229221 INFO  (jetty-launcher-27550-thread-3) [n:127.0.0.1:50095_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229221 INFO  (jetty-launcher-27550-thread-7) [n:127.0.0.1:45361_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229221 INFO  (jetty-launcher-27550-thread-4) [n:127.0.0.1:54890_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54890_solr
   [junit4]   2> 3229222 INFO  (jetty-launcher-27550-thread-6) [n:127.0.0.1:40978_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229222 INFO  (jetty-launcher-27550-thread-9) [n:127.0.0.1:42713_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229223 INFO  (jetty-launcher-27550-thread-8) [n:127.0.0.1:46589_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46589_solr
   [junit4]   2> 3229224 INFO  (jetty-launcher-27550-thread-3) [n:127.0.0.1:50095_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50095_solr
   [junit4]   2> 3229224 INFO  (jetty-launcher-27550-thread-4) [n:127.0.0.1:54890_solr    ] o.a.s.c.Overseer Overseer (id=98176448280461327-127.0.0.1:54890_solr-n_0000000000) starting
   [junit4]   2> 3229225 INFO  (jetty-launcher-27550-thread-7) [n:127.0.0.1:45361_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45361_solr
   [junit4]   2> 3229225 INFO  (jetty-launcher-27550-thread-1) [n:127.0.0.1:42835_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42835_solr
   [junit4]   2> 3229225 INFO  (jetty-launcher-27550-thread-6) [n:127.0.0.1:40978_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40978_solr
   [junit4]   2> 3229226 INFO  (jetty-launcher-27550-thread-2) [n:127.0.0.1:43223_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43223_solr
   [junit4]   2> 3229227 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229228 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229228 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229229 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229234 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229235 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229235 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229236 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3229237 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 3229238 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229238 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229239 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229239 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229239 INFO  (jetty-launcher-27550-thread-9) [n:127.0.0.1:42713_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42713_solr
   [junit4]   2> 3229241 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229246 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229246 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (5)
   [junit4]   2> 3229248 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229254 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 3229264 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229264 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229265 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229265 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229265 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229265 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229265 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229265 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 3229270 INFO  (jetty-launcher-27550-thread-10) [n:127.0.0.1:33640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 3229273 INFO  (jetty-launcher-27550-thread-10) [n:127.0.0.1:33640_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229278 INFO  (jetty-launcher-27550-thread-5) [n:127.0.0.1:49436_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 3229279 INFO  (jetty-launcher-27550-thread-10) [n:127.0.0.1:33640_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33640_solr
   [junit4]   2> 3229281 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229281 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229281 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229281 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229281 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229281 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229282 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229282 INFO  (jetty-launcher-27550-thread-5) [n:127.0.0.1:49436_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3229284 INFO  (jetty-launcher-27550-thread-5) [n:127.0.0.1:49436_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49436_solr
   [junit4]   2> 3229286 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 3229286 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229286 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 3229334 INFO  (zkCallback-27610-thread-1-processing-n:127.0.0.1:49436_solr) [n:127.0.0.1:49436_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 3229357 INFO  (jetty-launcher-27550-thread-1) [n:127.0.0.1:42835_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node1/.
   [junit4]   2> 3229358 INFO  (jetty-launcher-27550-thread-2) [n:127.0.0.1:43223_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node2/.
   [junit4]   2> 3229369 INFO  (jetty-launcher-27550-thread-9) [n:127.0.0.1:42713_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node9/.
   [junit4]   2> 3229374 INFO  (zkCallback-27604-thread-1-processing-n:127.0.0.1:33640_solr) [n:127.0.0.1:33640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 3229375 INFO  (jetty-launcher-27550-thread-4) [n:127.0.0.1:54890_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54890_solr
   [junit4]   2> 3229377 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229377 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229378 INFO  (zkCallback-27604-thread-1-processing-n:127.0.0.1:33640_solr) [n:127.0.0.1:33640_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229378 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229378 INFO  (zkCallback-27574-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229378 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229378 INFO  (zkCallback-27610-thread-1-processing-n:127.0.0.1:49436_solr) [n:127.0.0.1:49436_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229377 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229382 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229377 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 3229411 INFO  (jetty-launcher-27550-thread-7) [n:127.0.0.1:45361_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node6/.
   [junit4]   2> 3229416 INFO  (jetty-launcher-27550-thread-3) [n:127.0.0.1:50095_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node3/.
   [junit4]   2> 3229431 INFO  (jetty-launcher-27550-thread-6) [n:127.0.0.1:40978_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node8/.
   [junit4]   2> 3229453 INFO  (jetty-launcher-27550-thread-8) [n:127.0.0.1:46589_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node7/.
   [junit4]   2> 3229453 INFO  (jetty-launcher-27550-thread-10) [n:127.0.0.1:33640_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node10/.
   [junit4]   2> 3229455 INFO  (jetty-launcher-27550-thread-5) [n:127.0.0.1:49436_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node5/.
   [junit4]   2> 3229486 INFO  (jetty-launcher-27550-thread-4) [n:127.0.0.1:54890_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node4/.
   [junit4]   2> 3229498 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 3229499 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33933/solr ready
   [junit4]   2> 3229921 INFO  (qtp1820385743-164668) [n:127.0.0.1:50095_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&name=test_col&action=CREATE&numShards=5&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3229924 INFO  (OverseerThreadFactory-10652-thread-1-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 3230039 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard5_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard5&wt=javabin&version=2
   [junit4]   2> 3230039 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard3_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard3&wt=javabin&version=2
   [junit4]   2> 3230039 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230039 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230043 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard2_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard2&wt=javabin&version=2
   [junit4]   2> 3230043 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230045 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard2_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard2&wt=javabin&version=2
   [junit4]   2> 3230046 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard4_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard4&wt=javabin&version=2
   [junit4]   2> 3230046 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230046 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230047 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard3_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard3&wt=javabin&version=2
   [junit4]   2> 3230048 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230050 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard5_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard5&wt=javabin&version=2
   [junit4]   2> 3230051 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230054 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard4_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard4&wt=javabin&version=2
   [junit4]   2> 3230054 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230056 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard1_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard1&wt=javabin&version=2
   [junit4]   2> 3230056 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230059 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard1_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard1&wt=javabin&version=2
   [junit4]   2> 3230059 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3230164 INFO  (zkCallback-27571-thread-1-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27610-thread-1-processing-n:127.0.0.1:49436_solr) [n:127.0.0.1:49436_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27591-thread-1-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27597-thread-1-processing-n:127.0.0.1:46589_solr) [n:127.0.0.1:46589_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27572-thread-1-processing-n:127.0.0.1:43223_solr) [n:127.0.0.1:43223_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27574-thread-2-processing-n:127.0.0.1:54890_solr) [n:127.0.0.1:54890_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27589-thread-1-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27593-thread-1-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27604-thread-1-processing-n:127.0.0.1:33640_solr) [n:127.0.0.1:33640_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3230164 INFO  (zkCallback-27592-thread-1-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 3231056 WARN  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231066 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231070 WARN  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231071 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231082 WARN  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231083 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231086 WARN  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231087 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231092 WARN  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231092 WARN  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231104 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231106 WARN  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231107 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231108 WARN  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231109 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231109 WARN  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231110 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231111 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231114 WARN  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 3231115 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.s.IndexSchema [test_col_shard5_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 3231115 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3231119 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.s.IndexSchema [test_col_shard3_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 3231122 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231122 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard3_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231122 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231122 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.SolrCore [[test_col_shard3_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node5/test_col_shard3_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node5/./test_col_shard3_replica1/data/]
   [junit4]   2> 3231123 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231124 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231124 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard5_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231124 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231125 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.SolrCore [[test_col_shard5_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node3/test_col_shard5_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node3/./test_col_shard5_replica1/data/]
   [junit4]   2> 3231125 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231139 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231141 INFO  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231143 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.s.IndexSchema [test_col_shard4_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 3231144 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.s.IndexSchema [test_col_shard4_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 3231146 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231146 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard4_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231146 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231146 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.SolrCore [[test_col_shard4_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node8/test_col_shard4_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node8/./test_col_shard4_replica1/data/]
   [junit4]   2> 3231146 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231146 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard4_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231147 WARN  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231147 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231147 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231147 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.SolrCore [[test_col_shard4_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node2/test_col_shard4_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node2/./test_col_shard4_replica2/data/]
   [junit4]   2> 3231147 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231150 INFO  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231152 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.s.IndexSchema [test_col_shard1_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 3231154 INFO  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231156 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.s.IndexSchema [test_col_shard2_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 3231156 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.s.IndexSchema [test_col_shard5_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 3231156 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.s.IndexSchema [test_col_shard2_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 3231157 WARN  (qtp377448965-164657) [n:127.0.0.1:43223_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231159 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231159 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231159 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231159 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231159 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231159 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.SolrCore [[test_col_shard2_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node9/test_col_shard2_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node9/./test_col_shard2_replica2/data/]
   [junit4]   2> 3231160 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231160 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.SolrCore [[test_col_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node6/test_col_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node6/./test_col_shard1_replica2/data/]
   [junit4]   2> 3231160 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231160 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231161 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.s.IndexSchema [test_col_shard3_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 3231163 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231163 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard3_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231164 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231164 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.SolrCore [[test_col_shard3_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node10/test_col_shard3_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node10/./test_col_shard3_replica2/data/]
   [junit4]   2> 3231164 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231166 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231166 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard5_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231166 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.s.IndexSchema [test_col_shard1_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 3231166 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231166 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.SolrCore [[test_col_shard5_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node7/test_col_shard5_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node7/./test_col_shard5_replica2/data/]
   [junit4]   2> 3231167 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231168 INFO  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231169 INFO  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231169 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231169 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231169 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231169 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.SolrCore [[test_col_shard2_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node4/test_col_shard2_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node4/./test_col_shard2_replica1/data/]
   [junit4]   2> 3231170 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231170 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 3231170 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 3231170 INFO  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231170 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3231170 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.SolrCore [[test_col_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node1/test_col_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001/tempDir-001/node1/./test_col_shard1_replica1/data/]
   [junit4]   2> 3231171 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@221ce438
   [junit4]   2> 3231172 WARN  (qtp1615170909-164710) [n:127.0.0.1:49436_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231172 INFO  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231174 INFO  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231175 INFO  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=9.4501953125, floorSegmentMB=2.1025390625, forceMergeDeletesPctAllowed=7.337308201609734, segmentsPerTier=7.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3231216 WARN  (qtp1666312060-164758) [n:127.0.0.1:45361_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231218 WARN  (qtp327859697-164729) [n:127.0.0.1:40978_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231241 WARN  (qtp770200550-164679) [n:127.0.0.1:42835_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231267 WARN  (qtp585235591-164713) [n:127.0.0.1:46589_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231288 WARN  (qtp1322086139-164694) [n:127.0.0.1:54890_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231295 WARN  (qtp1756762061-164744) [n:127.0.0.1:42713_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231322 WARN  (qtp292245253-164734) [n:127.0.0.1:33640_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3231349 INFO  (qtp1820385743-164667) [n:127.0.0.1:50095_solr c:test_col s:shard5  x:test_col_shard5_repl

[...truncated too long message...]

able to read additional data from client sessionid 0x15ccaf5a511000d, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3331450 INFO  (zkCallback-27593-thread-2-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 3331457 INFO  (jetty-closer-27551-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5e0d2059{/solr,null,UNAVAILABLE}
   [junit4]   2> 3331459 INFO  (zkCallback-27593-thread-3-processing-n:127.0.0.1:42835_solr) [n:127.0.0.1:42835_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42835_solr
   [junit4]   2> 3332461 WARN  (zkCallback-27571-thread-2-processing-n:127.0.0.1:50095_solr) [n:127.0.0.1:50095_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3332462 INFO  (jetty-closer-27551-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4f636dae{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332678 WARN  (zkCallback-27589-thread-2-processing-n:127.0.0.1:40978_solr) [n:127.0.0.1:40978_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3332679 INFO  (jetty-closer-27551-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7cac8bd3{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332787 WARN  (zkCallback-27592-thread-2-processing-n:127.0.0.1:45361_solr) [n:127.0.0.1:45361_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3332788 INFO  (jetty-closer-27551-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@29efadfa{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332842 WARN  (zkCallback-27610-thread-2-processing-n:127.0.0.1:49436_solr) [n:127.0.0.1:49436_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3332843 INFO  (jetty-closer-27551-thread-8) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@494d1c67{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332904 WARN  (zkCallback-27604-thread-3-processing-n:127.0.0.1:33640_solr) [n:127.0.0.1:33640_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3332906 INFO  (jetty-closer-27551-thread-9) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7607ee4{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332935 WARN  (zkCallback-27591-thread-2-processing-n:127.0.0.1:42713_solr) [n:127.0.0.1:42713_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3332936 INFO  (jetty-closer-27551-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@34f219cb{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332960 INFO  (jetty-closer-27551-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@135b21b{/solr,null,UNAVAILABLE}
   [junit4]   2> 3332961 ERROR (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3332962 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33933 33933
   [junit4]   2> 3333063 INFO  (Thread-61944) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33933 33933
   [junit4]   2> 3333064 WARN  (Thread-61944) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 11 /solr/aliases.json
   [junit4]   2> 11 /solr/clusterprops.json
   [junit4]   2> 10 /solr/configs/org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set
   [junit4]   2> 10 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 11 /solr/clusterstate.json
   [junit4]   2> 10 /solr/collections/test_col/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98176448280461339-127.0.0.1:46589_solr-n_0000000001
   [junit4]   2> 2 /solr/overseer_elect/election/98176448280461337-127.0.0.1:45361_solr-n_0000000005
   [junit4]   2> 2 /solr/overseer_elect/election/98176448280461327-127.0.0.1:54890_solr-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/98176448280461336-127.0.0.1:42835_solr-n_0000000004
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 11 /solr/live_nodes
   [junit4]   2> 11 /solr/collections
   [junit4]   2>
   [junit4]   2> 3333064 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[3CDFB00F0E4D1BDC]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_3CDFB00F0E4D1BDC-001
   [junit4]   2> Jun 21, 2017 2:03:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {long_stored_idx=FST50, long_dv_idx=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=Memory doPackFST= true), long_dv_stored_idx=PostingsFormat(name=Memory doPackFST= true)}, docValues:{long_stored_idx=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Lucene54), long_dv_idx=DocValuesFormat(name=Memory), long_dv_stored=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Memory), long_dv=DocValuesFormat(name=Direct), long_dv_stored_idx=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1496, maxMBSortInHeap=7.1760009551501724, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=es-UY, timezone=Brazil/West
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=162730696,total=533725184
   [junit4]   2> NOTE: All tests run in this JVM: [TestHttpShardHandlerFactory, XsltUpdateRequestHandlerTest, TestNonDefinedSimilarityFactory, BlockJoinFacetRandomTest, EnumFieldTest, TestHalfAndHalfDocValues, ZkNodePropsTest, TestPushWriter, TestInfoStreamLogging, UtilsToolTest, TestCSVLoader, IgnoreCommitOptimizeUpdateProcessorFactoryTest, HdfsBasicDistributedZk2Test, SolrSlf4jReporterTest, FastVectorHighlighterTest, TestStressRecovery, TestJoin, TestPHPSerializedResponseWriter, TestNumericTerms32, VersionInfoTest, BlobRepositoryCloudTest, TestDocumentBuilder, DistributedFacetExistsSmallTest, TestDocSet, QueryElevationComponentTest, TestPseudoReturnFields, TestXmlQParser, TestConfigSetsAPI, HdfsSyncSliceTest, TestCryptoKeys, BasicAuthIntegrationTest, RequiredFieldsTest, TestRemoteStreaming, TestSha256AuthenticationProvider, TestStressLiveNodes, TestComponentsName, SpatialRPTFieldTypeTest, TestAuthorizationFramework, TestSolrConfigHandlerConcurrent, DistributedFacetPivotSmallAdvancedTest, TestReplicationHandler, BasicZkTest, TestSlowCompositeReaderWrapper, CustomCollectionTest, SecurityConfHandlerTest, OverseerStatusTest, ConfigSetsAPITest, DistributedSpellCheckComponentTest, SolrCoreTest, TestJmxIntegration, DeleteNodeTest, NumericFieldsTest, TestFieldCacheSanityChecker, TestSearcherReuse, TestDistributedSearch, HdfsRecoverLeaseTest, BigEndianAscendingWordSerializerTest, CollectionReloadTest, SharedFSAutoReplicaFailoverTest, LeaderInitiatedRecoveryOnCommitTest, TestQueryWrapperFilter, SuggestComponentTest, TestExactSharedStatsCache, TestReloadAndDeleteDocs, TestValueSourceCache, NodeMutatorTest, LargeFieldTest, SolrGangliaReporterTest, TestSSLRandomization, TestUniqueKeyFieldResource, PeerSyncTest, TestIBSimilarityFactory, SSLMigrationTest, TestHighlightDedupGrouping, TestAuthenticationFramework, SyncSliceTest, TestSolrQueryResponse, DeleteStatusTest, UniqFieldsUpdateProcessorFactoryTest, ResourceLoaderTest, TestInPlaceUpdatesDistrib, SortSpecParsingTest, TestReplicationHandlerBackup, TestSerializedLuceneMatchVersion, ScriptEngineTest, ClusterStateTest, TestRandomFlRTGCloud, FileBasedSpellCheckerTest, TestSolr4Spatial, TestFunctionQuery, ChangedSchemaMergeTest, TestClusterStateMutator, TestSolr4Spatial2, AnalyticsMergeStrategyTest, TestRandomFaceting, ReturnFieldsTest, ExternalFileFieldSortTest, TestSolrCloudWithHadoopAuthPlugin, ConfigureRecoveryStrategyTest, TestManagedSchemaThreadSafety, DistributedQueryElevationComponentTest, MissingSegmentRecoveryTest, LeaderElectionContextKeyTest, TestCloudRecovery, TestHighFrequencyDictionaryFactory, SimpleMLTQParserTest, CoreAdminCreateDiscoverTest, TestFieldCacheVsDocValues, BasicDistributedZkTest, LeaderElectionIntegrationTest, OverseerCollectionConfigSetProcessorTest, TestLocalFSCloudBackupRestore, TestOverriddenPrefixQueryForCustomFieldType, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestLeaderElectionZkExpiry, TestOnReconnectListenerSupport, TestRandomRequestDistribution, TestReplicaProperties, TestSizeLimitedDistributedMap, TestStressCloudBlindAtomicUpdates]
   [junit4] Completed [544/715 (1!)] on J0 in 104.89s, 5 tests, 1 error <<< FAILURES!

[...truncated 43993 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 378 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/378/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)  at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)  at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:153)  at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:110)  at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)  at sun.reflect.GeneratedConstructorAccessor220.newInstance(Unknown Source)  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:760)  at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:822)  at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:947)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:930)  at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)
        at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:153)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:110)
        at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
        at sun.reflect.GeneratedConstructorAccessor220.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:760)
        at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:822)
        at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:947)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:930)
        at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([3940C40812A9F8EE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
Repeated retry for 'inc' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@6d0fe77 (leader); Never got numFound=578; results=> {numFound=0,start=0,docs=[]}

Stack Trace:
java.lang.AssertionError: Repeated retry for 'inc' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@6d0fe77 (leader); Never got numFound=578; results=> {numFound=0,start=0,docs=[]}
        at __randomizedtesting.SeedInfo.seed([3940C40812A9F8EE:B114FBD2BC559516]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:457)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:388)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:154)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11606 lines...]
   [junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/init-core-data-001
   [junit4]   2> 209659 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 209659 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 209661 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 209662 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /pn_/f
   [junit4]   2> 209663 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 209664 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 209692 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 209720 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
   [junit4]   2> 209724 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 209729 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 209766 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 209772 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 209773 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 209775 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 209805 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 209818 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 209822 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 209822 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 209824 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 209824 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 209824 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/init-core-data-001/]
   [junit4]   2> 209825 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 209851 WARN  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 209924 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 209924 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 209925 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 209925 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 209932 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@29bc3791[collection1] main]
   [junit4]   2> 209933 WARN  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 209933 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 209934 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 209935 INFO  (searcherExecutor-196-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@29bc3791[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 209936 INFO  (coreLoadExecutor-195-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570908507051393024
   [junit4]   2> 209939 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 209943 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 209945 INFO  (Thread-395) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 209945 INFO  (Thread-395) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 209951 ERROR (Thread-395) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 210045 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ZkTestServer start zk server on port:46433
   [junit4]   2> 210062 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 210064 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
   [junit4]   2> 210067 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 210070 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 210072 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 210073 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 210075 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 210077 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 210079 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 210080 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 210082 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 210215 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/control-001/cores/collection1
   [junit4]   2> 210220 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 210227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@331c98b2{/pn_/f,null,AVAILABLE}
   [junit4]   2> 210230 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5cd45a05{SSL,[ssl, http/1.1]}{127.0.0.1:55828}
   [junit4]   2> 210230 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server Started @213131ms
   [junit4]   2> 210230 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/tempDir-001/control/data, hostContext=/pn_/f, hostPort=55828, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/control-001/cores}
   [junit4]   2> 210230 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 210230 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 210231 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 210231 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 210231 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T12:38:16.894Z
   [junit4]   2> 210240 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 210240 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/control-001/solr.xml
   [junit4]   2> 210248 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 210249 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46433/solr
   [junit4]   2> 210291 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 210292 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55828_pn_%2Ff
   [junit4]   2> 210293 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.Overseer Overseer (id=98181781691432965-127.0.0.1:55828_pn_%2Ff-n_0000000000) starting
   [junit4]   2> 210306 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55828_pn_%2Ff
   [junit4]   2> 210309 INFO  (zkCallback-70-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 210399 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/control-001/cores
   [junit4]   2> 210399 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 210400 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 210408 INFO  (OverseerStateUpdate-98181781691432965-127.0.0.1:55828_pn_%2Ff-n_0000000000) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 211429 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 211462 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 211466 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 211466 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 211466 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 211466 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/control-001/cores/collection1/data/]
   [junit4]   2> 211467 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 211484 WARN  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 211548 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 211548 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 211549 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 211549 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 211552 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5bfd013c[collection1] main]
   [junit4]   2> 211553 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 211554 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 211554 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 211558 INFO  (searcherExecutor-209-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5bfd013c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 211559 INFO  (coreLoadExecutor-208-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570908508753231872
   [junit4]   2> 211569 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 211569 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 211569 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:55828/pn_/f/collection1/
   [junit4]   2> 211569 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 211570 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:55828/pn_/f/collection1/ has no replicas
   [junit4]   2> 211570 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 211576 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:55828/pn_/f/collection1/ shard1
   [junit4]   2> 211728 INFO  (coreZkRegister-201-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff x:collection1 c:control_collection) [n:127.0.0.1:55828_pn_%2Ff c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 211933 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 211934 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46433/solr ready
   [junit4]   2> 211935 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 212052 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001/cores/collection1
   [junit4]   2> 212053 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001
   [junit4]   2> 212056 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 212058 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7bb8a734{/pn_/f,null,AVAILABLE}
   [junit4]   2> 212059 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@22f9823d{SSL,[ssl, http/1.1]}{127.0.0.1:47274}
   [junit4]   2> 212059 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server Started @214960ms
   [junit4]   2> 212059 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/pn_/f, hostPort=47274, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001/cores}
   [junit4]   2> 212059 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 212060 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 212060 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 212060 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 212060 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T12:38:18.723Z
   [junit4]   2> 212066 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 212067 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001/solr.xml
   [junit4]   2> 212077 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 212078 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46433/solr
   [junit4]   2> 212100 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 212111 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 212114 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47274_pn_%2Ff
   [junit4]   2> 212117 INFO  (zkCallback-70-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 212125 INFO  (zkCallback-74-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 212126 INFO  (zkCallback-79-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 212185 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001/cores
   [junit4]   2> 212185 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 212186 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 212190 INFO  (OverseerStateUpdate-98181781691432965-127.0.0.1:55828_pn_%2Ff-n_0000000000) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 213211 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 213224 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 213229 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 213229 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 213229 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 213229 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 213229 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 213245 WARN  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 213297 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 213297 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 213298 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 213298 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 213300 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6dd5d425[collection1] main]
   [junit4]   2> 213302 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 213303 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 213303 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 213306 INFO  (searcherExecutor-220-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6dd5d425[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 213307 INFO  (coreLoadExecutor-219-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570908510586142720
   [junit4]   2> 213315 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 213315 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 213315 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:47274/pn_/f/collection1/
   [junit4]   2> 213315 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 213315 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:47274/pn_/f/collection1/ has no replicas
   [junit4]   2> 213315 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 213320 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:47274/pn_/f/collection1/ shard1
   [junit4]   2> 213472 INFO  (coreZkRegister-214-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 213833 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001/cores/collection1
   [junit4]   2> 213833 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001
   [junit4]   2> 213836 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 213839 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1733069{/pn_/f,null,AVAILABLE}
   [junit4]   2> 213840 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@773f1a26{SSL,[ssl, http/1.1]}{127.0.0.1:46371}
   [junit4]   2> 213840 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server Started @216741ms
   [junit4]   2> 213840 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/pn_/f, hostPort=46371, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001/cores}
   [junit4]   2> 213840 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 213841 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 213841 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 213841 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 213841 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T12:38:20.504Z
   [junit4]   2> 213855 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 213856 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001/solr.xml
   [junit4]   2> 213864 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 213865 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46433/solr
   [junit4]   2> 213884 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 213889 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 213892 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46371_pn_%2Ff
   [junit4]   2> 213895 INFO  (zkCallback-74-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 213895 INFO  (zkCallback-79-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 213895 INFO  (zkCallback-70-thread-1-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 213896 INFO  (zkCallback-85-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 213981 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001/cores
   [junit4]   2> 213981 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 213982 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 213986 INFO  (OverseerStateUpdate-98181781691432965-127.0.0.1:55828_pn_%2Ff-n_0000000000) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 215002 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 215015 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 215019 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 215019 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 215020 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 215020 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 215020 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 215039 WARN  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 215090 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 215090 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 215091 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 215091 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 215093 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@23fdbc7d[collection1] main]
   [junit4]   2> 215095 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 215095 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 215095 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 215098 INFO  (searcherExecutor-231-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@23fdbc7d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 215099 INFO  (coreLoadExecutor-230-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570908512465190912
   [junit4]   2> 215104 INFO  (coreZkRegister-225-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 215105 INFO  (updateExecutor-82-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 215105 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 215106 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 215106 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 215106 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 215106 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:47274/pn_/f/collection1/] and I am [https://127.0.0.1:46371/pn_/f/collection1/]
   [junit4]   2> 215109 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:47274/pn_/f]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:46371_pn_%252Ff&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 215170 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 215171 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 215172 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:46371_pn_%2Ff, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:46371/pn_/f","node_name":"127.0.0.1:46371_pn_%2Ff","state":"down"}
   [junit4]   2> 215763 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001/cores/collection1
   [junit4]   2> 215764 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001
   [junit4]   2> 215766 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 215769 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e58d358{/pn_/f,null,AVAILABLE}
   [junit4]   2> 215769 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6cb67438{SSL,[ssl, http/1.1]}{127.0.0.1:57700}
   [junit4]   2> 215769 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.e.j.s.Server Started @218671ms
   [junit4]   2> 215770 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/pn_/f, hostPort=57700, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001/cores}
   [junit4]   2> 215770 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 215771 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 215771 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 215771 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 215771 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-22T12:38:22.434Z
   [junit4]   2> 215781 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 215781 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001/solr.xml
   [junit4]   2> 215801 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 215802 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46433/solr
   [junit4]   2> 215829 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:57700_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 215833 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:57700_pn_%2Ff    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 215836 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:57700_pn_%2Ff    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57700_pn_%2Ff
   [junit4]   2> 215837 INFO  (zkCallback-85-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff) [n:127.0.0.1:46371_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 215838 INFO  (zkCallback-79-thread-1-processing-n:127.0.0.1:47274_pn_%2Ff) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 215838 INFO  (zkCallback-74-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 215839 INFO  (zkCallback-70-thread-2-processing-n:127.0.0.1:55828_pn_%2Ff) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 215840 INFO  (zkCallback-92-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 215940 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:57700_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001/cores
   [junit4]   2> 215940 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [n:127.0.0.1:57700_pn_%2Ff    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 215940 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 215942 INFO  (OverseerStateUpdate-98181781691432965-127.0.0.1:55828_pn_%2Ff-n_0000000000) [n:127.0.0.1:55828_pn_%2Ff    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 216172 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:46371_pn_%2Ff, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:46371/pn_/f","node_name":"127.0.0.1:46371_pn_%2Ff","state":"recovering"}
   [junit4]   2> 216172 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 216172 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:46371_pn_%252Ff&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
   [junit4]   2> 216673 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:47274/pn_/f/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 216674 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:46371/pn_/f START replicas=[https://127.0.0.1:47274/pn_/f/collection1/] nUpdates=100
   [junit4]   2> 216687 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 216687 INFO  (qtp787514995-726) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/pn_/f path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 216688 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 216688 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 216688 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 216688 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 216689 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 216689 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 216689 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 216689 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 216689 INFO  (recoveryExecutor-83-thread-1-processing-n:127.0.0.1:46371_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:46371_pn_%2Ff c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 216962 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 216975 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 216980 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 216980 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 216980 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 216980 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_3940C40812A9F8EE-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 216980 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7255253f
   [junit4]   2> 216996 WARN  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 217067 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 217067 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 217068 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 217068 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 217070 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@19a914be[collection1] main]
   [junit4]   2> 217072 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 217072 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 217072 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 217075 INFO  (searcherExecutor-242-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@19a914be[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 217076 INFO  (coreLoadExecutor-241-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff) [n:127.0.0.1:57700_pn_%2Ff c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1570908514538225664
   [junit4]   2> 217083 INFO  (coreZkRegister-236-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 c:collection1) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 217084 INFO  (updateExecutor-89-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 217084 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 217085 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 217085 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 217085 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 217085 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:47274/pn_/f/collection1/] and I am [https://127.0.0.1:57700/pn_/f/collection1/]
   [junit4]   2> 217090 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:47274/pn_/f]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:57700_pn_%252Ff&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 217101 INFO  (qtp787514995-720) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 217102 INFO  (qtp787514995-720) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 217102 INFO  (qtp787514995-720) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:57700_pn_%2Ff, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:57700/pn_/f","node_name":"127.0.0.1:57700_pn_%2Ff","state":"down"}
   [junit4]   2> 217460 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 217460 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[3940C40812A9F8EE]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   1> replica:core_node1 rstate:active live:true
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node3 rstate:recovering live:true
   [junit4]   2> 218102 INFO  (qtp787514995-720) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:57700_pn_%2Ff, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:57700/pn_/f","node_name":"127.0.0.1:57700_pn_%2Ff","state":"recovering"}
   [junit4]   2> 218102 INFO  (qtp787514995-720) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 218102 INFO  (qtp787514995-720) [n:127.0.0.1:47274_pn_%2Ff    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:57700_pn_%252Ff&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
   [junit4]   1> -
   [junit4]   1> replica:core_node1 rstate:active live:true
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node3 rstate:recovering live:true
   [junit4]   2> 218603 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:47274/pn_/f/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 218603 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:57700/pn_/f START replicas=[https://127.0.0.1:47274/pn_/f/collection1/] nUpdates=100
   [junit4]   2> 218614 INFO  (qtp787514995-723) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 218614 INFO  (qtp787514995-723) [n:127.0.0.1:47274_pn_%2Ff c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/pn_/f path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 218615 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 218615 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 218615 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 218615 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 218616 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 218616 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 218616 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 218616 INFO  (recoveryExecutor-90-thread-1-processing-n:127.0.0.1:57700_pn_%2Ff x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:57700_pn_%2Ff c:collectio

[...truncated too long message...]

otsearch.randomizedtesting.*]
   [junit4]   2>     java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> JUnit4-serializer-daemon:
   [junit4]   2>     java.lang.Thread.sleep(Native Method)
   [junit4]   2>     com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>
   [junit4]   2> IPC Parameter Sending Thread #2:
   [junit4]   2>     [...sun.*]
   [junit4]   2>     java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>     java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2>     java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]   2>     java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]   2>     java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]   2>     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>     java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> Thread-56506:
   [junit4]   2>     java.lang.Thread.sleep(Native Method)
   [junit4]   2>     org.apache.zookeeper.Login$1.run(Login.java:185)
   [junit4]   2>     java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> SUITE-HdfsRecoveryZkTest-seed#[3940C40812A9F8EE]:
   [junit4]   2>     java.lang.Object.wait(Native Method)
   [junit4]   2>     java.lang.Thread.join(Thread.java:1260)
   [junit4]   2>     [...com.carrotsearch.randomizedtesting.*]
   [junit4]   2>
   [junit4]   2> 3619000 INFO  (SUITE-HdfsRecoveryZkTest-seed#[3940C40812A9F8EE]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {rnd_b=FSTOrd50, _version_=BlockTreeOrds(blocksize=128), a_t=PostingsFormat(name=Memory doPackFST= true), a_i=FSTOrd50, id=FSTOrd50}, docValues:{}, maxPointsInLeafNode=812, maxMBSortInHeap=7.660547897643366, sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=el-GR, timezone=Europe/Dublin
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=173527552,total=524288000
   [junit4]   2> NOTE: All tests run in this JVM: [TestRandomFlRTGCloud, SuggestComponentTest, DistributedQueryElevationComponentTest, TestLeaderElectionWithEmptyReplica, TestManagedResourceStorage, TestInPlaceUpdatesDistrib, ClassificationUpdateProcessorTest, TestNumericTerms32, SparseHLLTest, QueryEqualityTest, OutputWriterTest, DateFieldTest, TestHttpShardHandlerFactory, SolrCoreTest, TestUniqueKeyFieldResource, TestFunctionQuery, TestFieldCacheSanityChecker, BasicZkTest, RegexBoostProcessorTest, NotRequiredUniqueKeyTest, TestSlowCompositeReaderWrapper, AtomicUpdatesTest, ShufflingReplicaListTransformerTest, TestCryptoKeys, PathHierarchyTokenizerFactoryTest, EnumFieldTest, TestSolrCloudSnapshots, TestJettySolrRunner, ReplicationFactorTest, JsonValidatorTest, HdfsBasicDistributedZk2Test, DocValuesNotIndexedTest, TestBadConfig, CursorPagingTest, TestSolr4Spatial2, SolrGangliaReporterTest, CoreAdminCreateDiscoverTest, HdfsRecoverLeaseTest, TestHighFrequencyDictionaryFactory, TestManagedSchemaThreadSafety, SortSpecParsingTest, TestCloudNestedDocsSort, TestCrossCoreJoin, DistributedFacetExistsSmallTest, SolrMetricManagerTest, StandardRequestHandlerTest, FacetPivotSmallTest, TestHighlightDedupGrouping, TestScoreJoinQPScore, TestSortByMinMaxFunction, PolyFieldTest, TestNonDefinedSimilarityFactory, TestGeoJSONResponseWriter, TestSolrConfigHandler, TestReRankQParserPlugin, SolrCLIZkUtilsTest, TestFieldCacheVsDocValues, TestLocalFSCloudBackupRestore, TestBackupRepositoryFactory, FullHLLTest, CustomCollectionTest, DataDrivenBlockJoinTest, TestIBSimilarityFactory, TestCoreAdminApis, TestDocTermOrds, TestXmlQParser, TestStandardQParsers, SolrSlf4jReporterTest, TestValueSourceCache, TestSchemaResource, BigEndianAscendingWordSerializerTest, TestSolrQueryResponse, AnalyticsMergeStrategyTest, TestPHPSerializedResponseWriter, RecoveryAfterSoftCommitTest, LeaderInitiatedRecoveryOnCommitTest, LeaderElectionContextKeyTest, TestManagedSynonymGraphFilterFactory, AddBlockUpdateTest, CurrencyFieldOpenExchangeTest, ClusterStateTest, TestSQLHandler, TestInfoStreamLogging, OverseerStatusTest, TestExactSharedStatsCache, TestSSLRandomization, TestComponentsName, BufferStoreTest, TestCloudJSONFacetJoinDomain, TestQueryWrapperFilter, TestSubQueryTransformerCrossCore, TestRandomFaceting, TestComplexPhraseLeadingWildcard, TestDistributedSearch, TestSolrCoreProperties, TestDocSet, JavabinLoaderTest, HighlighterConfigTest, TestJsonFacets, AliasIntegrationTest, LeaderElectionIntegrationTest, TestStressRecovery, PeerSyncTest, TestRandomDVFaceting, TestHashQParserPlugin, TestRealTimeGet, QueryElevationComponentTest, TestMacros, TestFastLRUCache, DisMaxRequestHandlerTest, TestCloudRecovery, BasicDistributedZkTest, SyncSliceTest, OverseerTest, ShardRoutingTest, RecoveryZkTest, TestReplicationHandler, TestHashPartitioner, TestJoin, HardAutoCommitTest, TestSolr4Spatial, SpellCheckComponentTest, TestGroupingSearch, ConvertedLegacyTest, TestFiltering, DirectUpdateHandlerTest, HighlighterTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SolrRequestParserTest, SuggesterTSTTest, TestStressLucene, TestTrie, TestCSVLoader, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, TestRemoteStreaming, DistanceFunctionTest, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, CacheHeaderTest, TestQueryUtils, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestOmitPositions, RequestHandlersTest, RequiredFieldsTest, FastVectorHighlighterTest, LoggingHandlerTest, IndexSchemaTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, QueryParsingTest, TestAnalyzedSuggestions, SearchHandlerTest, TestLFUCache, PrimUtilsTest, DOMUtilTest, TestSolrJ, TestDocumentBuilder, ZkNodePropsTest, UUIDFieldTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestSimpleTrackingShardHandler, AssignTest, BaseCdcrDistributedZkTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, ConcurrentDeleteAndCreateCollectionTest, DistribCursorPagingTest, MoveReplicaTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerRolesTest, SSLMigrationTest, SaslZkACLProviderTest, ShardSplitTest, TriLevelCompositeIdRoutingTest, HdfsRecoveryZkTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsRecoveryZkTest -Dtests.seed=3940C40812A9F8EE -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt -Dtests.locale=el-GR -Dtests.timezone=Europe/Dublin -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | HdfsRecoveryZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog
   [junit4]    > at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)
   [junit4]    > at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
   [junit4]    > at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:153)
   [junit4]    > at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:110)
   [junit4]    > at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
   [junit4]    > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4]    > at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:760)
   [junit4]    > at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:822)
   [junit4]    > at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
   [junit4]    > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:947)
   [junit4]    > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:830)
   [junit4]    > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:930)
   [junit4]    > at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
   [junit4]    > at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([3940C40812A9F8EE]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [542/715 (2!)] on J0 in 39.40s, 1 test, 1 failure <<< FAILURES!

[...truncated 43989 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 379 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/379/

3 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at <a href="http://127.0.0.1:53569:">http://127.0.0.1:53569: Error CREATEing SolrCore 'test_unload_shard_and_collection_2': Unable to create core [test_unload_shard_and_collection_2] Caused by: Direct buffer memory

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:53569:">http://127.0.0.1:53569: Error CREATEing SolrCore 'test_unload_shard_and_collection_2': Unable to create core [test_unload_shard_and_collection_2] Caused by: Direct buffer memory
        at __randomizedtesting.SeedInfo.seed([59E7ECE3D34C14EF:D1B3D3397DB07917]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:594)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:133)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:70)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1498240318229,generation=2,filelist=[_rhc.fdt, _rhc.fdx, _rhc.fnm, _rhc.nvd, _rhc.nvm, _rhc.si, _rhc_Lucene50_0.doc, _rhc_Lucene50_0.tim, _rhc_Lucene50_0.tip, _rib.fdt, _rib.fdx, _rib.fnm, _rib.nvd, _rib.nvm, _rib.si, _rib_Lucene50_0.doc, _rib_Lucene50_0.tim, _rib_Lucene50_0.tip, _ric.cfe, _ric.cfs, _ric.si, _rid.cfe, _rid.cfs, _rid.si, _rie.cfe, _rie.cfs, _rie.si, _rjz.cfe, _rjz.cfs, _rjz.si, _rk0.fdt, _rk0.fdx, _rk0.fnm, _rk0.nvd, _rk0.nvm, _rk0.si, _rk0_Lucene50_0.doc, _rk0_Lucene50_0.tim, _rk0_Lucene50_0.tip, _rk1.cfe, _rk1.cfs, _rk1.si, segments_2]}]> but was:<[{indexVersion=1498240318229,generation=2,filelist=[_rhc.fdt, _rhc.fdx, _rhc.fnm, _rhc.nvd, _rhc.nvm, _rhc.si, _rhc_Lucene50_0.doc, _rhc_Lucene50_0.tim, _rhc_Lucene50_0.tip, _rib.fdt, _rib.fdx, _rib.fnm, _rib.nvd, _rib.nvm, _rib.si, _rib_Lucene50_0.doc, _rib_Lucene50_0.tim, _rib_Lucene50_0.tip, _ric.cfe, _ric.cfs, _ric.si, _rid.cfe, _rid.cfs, _rid.si, _rie.cfe, _rie.cfs, _rie.si, _rjz.cfe, _rjz.cfs, _rjz.si, _rk0.fdt, _rk0.fdx, _rk0.fnm, _rk0.nvd, _rk0.nvm, _rk0.si, _rk0_Lucene50_0.doc, _rk0_Lucene50_0.tim, _rk0_Lucene50_0.tip, _rk1.cfe, _rk1.cfs, _rk1.si, segments_2]}, {indexVersion=1498240318229,generation=3,filelist=[_rif.fdt, _rif.fdx, _rif.fnm, _rif.nvd, _rif.nvm, _rif.si, _rif_Lucene50_0.doc, _rif_Lucene50_0.tim, _rif_Lucene50_0.tip, _rjz.cfe, _rjz.cfs, _rjz.si, _rk0.fdt, _rk0.fdx, _rk0.fnm, _rk0.nvd, _rk0.nvm, _rk0.si, _rk0_Lucene50_0.doc, _rk0_Lucene50_0.tim, _rk0_Lucene50_0.tip, _rk1.cfe, _rk1.cfs, _rk1.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1498240318229,generation=2,filelist=[_rhc.fdt, _rhc.fdx, _rhc.fnm, _rhc.nvd, _rhc.nvm, _rhc.si, _rhc_Lucene50_0.doc, _rhc_Lucene50_0.tim, _rhc_Lucene50_0.tip, _rib.fdt, _rib.fdx, _rib.fnm, _rib.nvd, _rib.nvm, _rib.si, _rib_Lucene50_0.doc, _rib_Lucene50_0.tim, _rib_Lucene50_0.tip, _ric.cfe, _ric.cfs, _ric.si, _rid.cfe, _rid.cfs, _rid.si, _rie.cfe, _rie.cfs, _rie.si, _rjz.cfe, _rjz.cfs, _rjz.si, _rk0.fdt, _rk0.fdx, _rk0.fnm, _rk0.nvd, _rk0.nvm, _rk0.si, _rk0_Lucene50_0.doc, _rk0_Lucene50_0.tim, _rk0_Lucene50_0.tip, _rk1.cfe, _rk1.cfs, _rk1.si, segments_2]}]> but was:<[{indexVersion=1498240318229,generation=2,filelist=[_rhc.fdt, _rhc.fdx, _rhc.fnm, _rhc.nvd, _rhc.nvm, _rhc.si, _rhc_Lucene50_0.doc, _rhc_Lucene50_0.tim, _rhc_Lucene50_0.tip, _rib.fdt, _rib.fdx, _rib.fnm, _rib.nvd, _rib.nvm, _rib.si, _rib_Lucene50_0.doc, _rib_Lucene50_0.tim, _rib_Lucene50_0.tip, _ric.cfe, _ric.cfs, _ric.si, _rid.cfe, _rid.cfs, _rid.si, _rie.cfe, _rie.cfs, _rie.si, _rjz.cfe, _rjz.cfs, _rjz.si, _rk0.fdt, _rk0.fdx, _rk0.fnm, _rk0.nvd, _rk0.nvm, _rk0.si, _rk0_Lucene50_0.doc, _rk0_Lucene50_0.tim, _rk0_Lucene50_0.tip, _rk1.cfe, _rk1.cfs, _rk1.si, segments_2]}, {indexVersion=1498240318229,generation=3,filelist=[_rif.fdt, _rif.fdx, _rif.fnm, _rif.nvd, _rif.nvm, _rif.si, _rif_Lucene50_0.doc, _rif_Lucene50_0.tim, _rif_Lucene50_0.tip, _rjz.cfe, _rjz.cfs, _rjz.si, _rk0.fdt, _rk0.fdx, _rk0.fnm, _rk0.nvd, _rk0.nvm, _rk0.si, _rk0_Lucene50_0.doc, _rk0_Lucene50_0.tim, _rk0_Lucene50_0.tip, _rk1.cfe, _rk1.cfs, _rk1.si, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([59E7ECE3D34C14EF:7C30F7D3A3041AEC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1281)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testAliasHandling

Error Message:
Collection not found: testalias

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: testalias
        at __randomizedtesting.SeedInfo.seed([7CAF511051E022C3:79503EE092AE26E1]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1401)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1094)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testAliasHandling(CloudSolrClientTest.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11618 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/init-core-data-001
   [junit4]   2> 132542 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 132549 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 132551 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 132551 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 134399 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 136307 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 136941 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 136986 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 137135 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 137206 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_48422_hdfs____ijchei/webapp
   [junit4]   2> 139036 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:48422
   [junit4]   2> 142746 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 142754 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 142793 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_48845_datanode____gbak2k/webapp
   [junit4]   2> 144111 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:48845
   [junit4]   2> 145777 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 145779 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 145829 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45791_datanode____.5omr3f/webapp
   [junit4]   2> 147043 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[59E7ECE3D34C14EF]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45791
   [junit4]   2> 150339 INFO  (IPC Server handler 3 on 58987) [    ] BlockStateChange BLOCK* processReport: from storage DS-a976d70f-9fa6-4d4b-a62d-216d12d8dc92 node DatanodeRegistration(127.0.0.1:57598, datanodeUuid=37d7c5ad-a607-4632-87d4-641eace78bf4, infoPort=51348, infoSecurePort=0, ipcPort=37100, storageInfo=lv=-56;cid=testClusterID;nsid=1047746299;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 150339 INFO  (IPC Server handler 4 on 58987) [    ] BlockStateChange BLOCK* processReport: from storage DS-547e4948-b83e-42cc-ac66-efe1335dbd8c node DatanodeRegistration(127.0.0.1:59212, datanodeUuid=30734d1f-7d2a-4f52-9726-e70d60d91405, infoPort=34312, infoSecurePort=0, ipcPort=37215, storageInfo=lv=-56;cid=testClusterID;nsid=1047746299;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 150339 INFO  (IPC Server handler 4 on 58987) [    ] BlockStateChange BLOCK* processReport: from storage DS-c2a0f254-0f5b-496f-8b83-8dcbe8ae7346 node DatanodeRegistration(127.0.0.1:59212, datanodeUuid=30734d1f-7d2a-4f52-9726-e70d60d91405, infoPort=34312, infoSecurePort=0, ipcPort=37215, storageInfo=lv=-56;cid=testClusterID;nsid=1047746299;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 150349 INFO  (IPC Server handler 3 on 58987) [    ] BlockStateChange BLOCK* processReport: from storage DS-7519362b-7c93-4a53-9cab-f1065118ed3b node DatanodeRegistration(127.0.0.1:57598, datanodeUuid=37d7c5ad-a607-4632-87d4-641eace78bf4, infoPort=51348, infoSecurePort=0, ipcPort=37100, storageInfo=lv=-56;cid=testClusterID;nsid=1047746299;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 151566 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 151567 INFO  (Thread-304) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 151567 INFO  (Thread-304) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 151570 ERROR (Thread-304) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 151669 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ZkTestServer start zk server on port:35765
   [junit4]   2> 151778 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 151783 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 151785 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 151787 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 151788 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 151791 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 151793 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 151801 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 151802 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 151804 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 151805 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 152010 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/control-001/cores/collection1
   [junit4]   2> 152014 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 152027 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@284cc931{/,null,AVAILABLE}
   [junit4]   2> 152028 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@75ec1e8c{HTTP/1.1,[http/1.1]}{127.0.0.1:60086}
   [junit4]   2> 152028 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server Started @155193ms
   [junit4]   2> 152029 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:58987/hdfs__localhost_58987__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001_tempDir-002_control_data, hostContext=/, hostPort=60086, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/control-001/cores}
   [junit4]   2> 152029 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 152029 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 152029 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 152029 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 152029 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-23T17:13:49.897Z
   [junit4]   2> 152049 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 152050 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/control-001/solr.xml
   [junit4]   2> 152058 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 152060 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35765/solr
   [junit4]   2> 152143 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:60086_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 152143 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:60086_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60086_
   [junit4]   2> 152144 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:60086_    ] o.a.s.c.Overseer Overseer (id=98188527497117700-127.0.0.1:60086_-n_0000000000) starting
   [junit4]   2> 152184 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:60086_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60086_
   [junit4]   2> 152188 INFO  (zkCallback-128-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 152267 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:60086_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/control-001/cores
   [junit4]   2> 152267 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:60086_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 152268 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 152282 INFO  (OverseerStateUpdate-98188527497117700-127.0.0.1:60086_-n_0000000000) [n:127.0.0.1:60086_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 153337 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 153447 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 153769 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 153809 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 153829 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58987/solr_hdfs_home
   [junit4]   2> 153829 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 153829 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 153829 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/control-001/cores/collection1], dataDir=[hdfs://localhost:58987/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 153830 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7941551d
   [junit4]   2> 153831 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 153865 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 153865 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 153865 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 155847 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 155858 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 155970 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 156022 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 156023 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 156023 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 156074 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 156074 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 156465 INFO  (IPC Server handler 7 on 58987) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59212 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2a0f254-0f5b-496f-8b83-8dcbe8ae7346:NORMAL:127.0.0.1:59212|RBW], ReplicaUC[[DISK]DS-a976d70f-9fa6-4d4b-a62d-216d12d8dc92:NORMAL:127.0.0.1:57598|RBW]]} size 71
   [junit4]   2> 156471 INFO  (IPC Server handler 0 on 58987) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57598 is added to blk_1073741825_1001 size 71
   [junit4]   2> 156929 WARN  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 157012 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 157012 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 157012 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 157046 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 157046 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 157060 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=351509594, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 157263 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@77361b0e[collection1] main]
   [junit4]   2> 157265 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 157266 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 157268 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 157270 INFO  (searcherExecutor-715-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@77361b0e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 157295 INFO  (coreLoadExecutor-714-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571016445959077888
   [junit4]   2> 157351 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 157351 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 157351 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60086/collection1/
   [junit4]   2> 157351 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 157351 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:60086/collection1/ has no replicas
   [junit4]   2> 157351 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 157360 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60086/collection1/ shard1
   [junit4]   2> 157370 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 157371 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35765/solr ready
   [junit4]   2> 157375 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 157375 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 157461 INFO  (coreZkRegister-707-thread-1-processing-n:127.0.0.1:60086_ x:collection1 c:control_collection) [n:127.0.0.1:60086_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 157698 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-1-001/cores/collection1
   [junit4]   2> 157699 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-1-001
   [junit4]   2> 157700 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 157703 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3862a25f{/,null,AVAILABLE}
   [junit4]   2> 157706 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@50d9c28d{HTTP/1.1,[http/1.1]}{127.0.0.1:53569}
   [junit4]   2> 157707 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server Started @160871ms
   [junit4]   2> 157707 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:58987/hdfs__localhost_58987__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=53569, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-1-001/cores}
   [junit4]   2> 157710 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 157711 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 157711 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 157711 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 157711 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-23T17:13:55.579Z
   [junit4]   2> 157725 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 157726 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-1-001/solr.xml
   [junit4]   2> 157737 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 157738 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35765/solr
   [junit4]   2> 157791 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 157794 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:53569_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 157796 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:53569_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53569_
   [junit4]   2> 157799 INFO  (zkCallback-132-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 157799 INFO  (zkCallback-128-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 157800 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 157956 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:53569_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-1-001/cores
   [junit4]   2> 157956 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:53569_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 157968 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 157970 INFO  (OverseerStateUpdate-98188527497117700-127.0.0.1:60086_-n_0000000000) [n:127.0.0.1:60086_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 158073 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 158999 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 159019 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 159156 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 159188 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 159189 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58987/solr_hdfs_home
   [junit4]   2> 159189 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 159189 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 159189 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:58987/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 159189 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7941551d
   [junit4]   2> 159190 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 159200 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 159200 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 159200 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 159221 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 159223 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 159254 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 159263 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 159263 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 159263 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 159276 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 159277 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 159338 INFO  (IPC Server handler 9 on 58987) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59212 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a976d70f-9fa6-4d4b-a62d-216d12d8dc92:NORMAL:127.0.0.1:57598|RBW], ReplicaUC[[DISK]DS-547e4948-b83e-42cc-ac66-efe1335dbd8c:NORMAL:127.0.0.1:59212|RBW]]} size 0
   [junit4]   2> 159340 INFO  (IPC Server handler 7 on 58987) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57598 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-547e4948-b83e-42cc-ac66-efe1335dbd8c:NORMAL:127.0.0.1:59212|RBW], ReplicaUC[[DISK]DS-7519362b-7c93-4a53-9cab-f1065118ed3b:NORMAL:127.0.0.1:57598|FINALIZED]]} size 0
   [junit4]   2> 159373 WARN  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 159462 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 159462 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 159462 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 159484 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 159484 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 159495 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=351509594, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 159519 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@752c03c7[collection1] main]
   [junit4]   2> 159521 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 159521 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 159521 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 159527 INFO  (searcherExecutor-726-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@752c03c7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 159534 INFO  (coreLoadExecutor-725-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571016448306839552
   [junit4]   2> 159543 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 159543 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 159543 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53569/collection1/
   [junit4]   2> 159543 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 159543 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:53569/collection1/ has no replicas
   [junit4]   2> 159543 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 159549 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53569/collection1/ shard2
   [junit4]   2> 159652 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 159700 INFO  (coreZkRegister-720-thread-1-processing-n:127.0.0.1:53569_ x:collection1 c:collection1) [n:127.0.0.1:53569_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 159806 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 160098 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-2-001/cores/collection1
   [junit4]   2> 160098 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-2-001
   [junit4]   2> 160101 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 160103 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3201a7ee{/,null,AVAILABLE}
   [junit4]   2> 160103 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2b3b21{HTTP/1.1,[http/1.1]}{127.0.0.1:55385}
   [junit4]   2> 160104 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server Started @163268ms
   [junit4]   2> 160104 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:58987/hdfs__localhost_58987__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=55385, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-2-001/cores}
   [junit4]   2> 160105 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 160105 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 160105 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 160105 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 160105 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-23T17:13:57.973Z
   [junit4]   2> 160113 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 160113 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-2-001/solr.xml
   [junit4]   2> 160120 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 160122 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35765/solr
   [junit4]   2> 160141 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:55385_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 160144 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:55385_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 160147 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:55385_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55385_
   [junit4]   2> 160149 INFO  (zkCallback-132-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 160149 INFO  (zkCallback-128-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 160149 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 160161 INFO  (zkCallback-144-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 160244 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:55385_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-2-001/cores
   [junit4]   2> 160244 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:55385_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 160247 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 160251 INFO  (OverseerStateUpdate-98188527497117700-127.0.0.1:60086_-n_0000000000) [n:127.0.0.1:60086_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 160354 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 160354 INFO  (zkCallback-144-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 161266 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 161281 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 161397 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 161420 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 161421 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58987/solr_hdfs_home
   [junit4]   2> 161421 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 161421 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 161421 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:58987/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 161421 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7941551d
   [junit4]   2> 161422 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 161431 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 161431 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 161431 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 161444 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 161446 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 161483 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58987/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 161490 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 161490 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 161490 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 161503 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 161504 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 161557 INFO  (IPC Server handler 6 on 58987) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59212 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7519362b-7c93-4a53-9cab-f1065118ed3b:NORMAL:127.0.0.1:57598|RBW], ReplicaUC[[DISK]DS-c2a0f254-0f5b-496f-8b83-8dcbe8ae7346:NORMAL:127.0.0.1:59212|RBW]]} size 0
   [junit4]   2> 161559 INFO  (IPC Server handler 7 on 58987) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57598 is added to blk_1073741827_1003 size 71
   [junit4]   2> 161582 WARN  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 161658 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 161658 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 161658 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 161675 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 161675 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 161681 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=351509594, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 161700 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1f2d2c4f[collection1] main]
   [junit4]   2> 161701 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 161701 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 161701 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 161704 INFO  (searcherExecutor-737-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1f2d2c4f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 161711 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571016450589589504
   [junit4]   2> 161727 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 161727 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 161727 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55385/collection1/
   [junit4]   2> 161727 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 161727 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:55385/collection1/ has no replicas
   [junit4]   2> 161727 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 161733 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55385/collection1/ shard1
   [junit4]   2> 161836 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 161836 INFO  (zkCallback-144-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 161872 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-3-001/cores/collection1
   [junit4]   2> 161873 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-3-001
   [junit4]   2> 161874 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 161877 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@665694a1{/,null,AVAILABLE}
   [junit4]   2> 161877 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@520c275d{HTTP/1.1,[http/1.1]}{127.0.0.1:34435}
   [junit4]   2> 161877 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.e.j.s.Server Started @165042ms
   [junit4]   2> 161878 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:58987/hdfs__localhost_58987__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001_tempDir-002_jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=34435, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-3-001/cores}
   [junit4]   2> 161878 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 161879 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 161879 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 161879 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 161879 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-23T17:13:59.747Z
   [junit4]   2> 161884 INFO  (coreZkRegister-731-thread-1-processing-n:127.0.0.1:55385_ x:collection1 c:collection1) [n:127.0.0.1:55385_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 161884 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 161884 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-3-001/solr.xml
   [junit4]   2> 161910 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 161911 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35765/solr
   [junit4]   2> 161943 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:34435_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 161946 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:34435_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 161949 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:34435_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34435_
   [junit4]   2> 161950 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 161950 INFO  (zkCallback-132-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 161951 INFO  (zkCallback-128-thread-1-processing-n:127.0.0.1:60086_) [n:127.0.0.1:60086_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 161950 INFO  (zkCallback-144-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 161955 INFO  (zkCallback-150-thread-1-processing-n:127.0.0.1:34435_) [n:127.0.0.1:34435_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 162052 INFO  (zkCallback-138-thread-1-processing-n:127.0.0.1:53569_) [n:127.0.0.1:53569_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 162052 INFO  (zkCallback-144-thread-1-processing-n:127.0.0.1:55385_) [n:127.0.0.1:55385_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 162117 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:34435_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_59E7ECE3D34C14EF-001/shard-3-001/cores
   [junit4]   2> 162117 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[59E7ECE3D34C14EF]) [n:127.0.0.1:34435_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 162125 INFO  (co

[...truncated too long message...]

Coll.shard1.replica2
   [junit4]   2> 143348 INFO  (coreCloseExecutor-385-thread-4) [n:127.0.0.1:48656_solr c:localShardsTestColl s:shard2 r:core_node1 x:localShardsTestColl_shard2_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.localShardsTestColl.shard2.replica2
   [junit4]   2> 143348 INFO  (coreCloseExecutor-386-thread-4) [n:127.0.0.1:42704_solr c:multicollection2 s:shard1 r:core_node1 x:multicollection2_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.multicollection2.shard1.replica1
   [junit4]   2> 143355 INFO  (coreCloseExecutor-386-thread-3) [n:127.0.0.1:42704_solr c:multicollection1 s:shard2 r:core_node1 x:multicollection1_shard2_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.multicollection1.shard2.replica1
   [junit4]   2> 143367 INFO  (coreCloseExecutor-386-thread-5) [n:127.0.0.1:42704_solr c:localShardsTestColl s:shard1 r:core_node4 x:localShardsTestColl_shard1_replica3] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.localShardsTestColl.shard1.replica3
   [junit4]   2> 143380 INFO  (coreCloseExecutor-385-thread-2) [n:127.0.0.1:48656_solr c:localShardsTestColl s:shard3 r:core_node5 x:localShardsTestColl_shard3_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.localShardsTestColl.shard3.replica2
   [junit4]   2> 143399 INFO  (coreCloseExecutor-386-thread-6) [n:127.0.0.1:42704_solr c:localShardsTestColl s:shard2 r:core_node9 x:localShardsTestColl_shard2_replica3] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.localShardsTestColl.shard2.replica3
   [junit4]   2> 143399 INFO  (coreCloseExecutor-385-thread-1) [n:127.0.0.1:48656_solr c:foo s:shard1 r:core_node2 x:foo_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.foo.shard1.replica1
   [junit4]   2> 143400 INFO  (coreCloseExecutor-386-thread-2) [n:127.0.0.1:42704_solr c:foo s:shard1 r:core_node3 x:foo_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.foo.shard1.replica2
   [junit4]   2> 143409 WARN  (zkCallback-713-thread-1-processing-n:127.0.0.1:41764_solr) [n:127.0.0.1:41764_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 143409 INFO  (zkCallback-714-thread-1-processing-n:127.0.0.1:48656_solr) [n:127.0.0.1:48656_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 143409 INFO  (zkCallback-709-thread-1-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 143409 INFO  (coreCloseExecutor-386-thread-7) [n:127.0.0.1:42704_solr c:localShardsTestColl s:shard3 r:core_node3 x:localShardsTestColl_shard3_replica3] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.localShardsTestColl.shard3.replica3
   [junit4]   2> 143417 INFO  (zkCallback-714-thread-1-processing-n:127.0.0.1:48656_solr) [n:127.0.0.1:48656_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/foo/state.json] for collection [foo] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143424 INFO  (zkCallback-714-thread-3-processing-n:127.0.0.1:48656_solr) [n:127.0.0.1:48656_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/localShardsTestColl/state.json] for collection [localShardsTestColl] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143424 INFO  (zkCallback-709-thread-1-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143428 INFO  (zkCallback-709-thread-1-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/multicollection1/state.json] for collection [multicollection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143428 INFO  (zkCallback-709-thread-2-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/localShardsTestColl/state.json] for collection [localShardsTestColl] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143428 INFO  (zkCallback-709-thread-3-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/foo/state.json] for collection [foo] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143432 INFO  (zkCallback-709-thread-4-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/multicollection2/state.json] for collection [multicollection2] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 143432 INFO  (jetty-closer-698-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@59a29614{/solr,null,UNAVAILABLE}
   [junit4]   2> 143435 INFO  (jetty-closer-698-thread-3) [    ] o.a.s.c.Overseer Overseer (id=98188836055023625-127.0.0.1:48656_solr-n_0000000000) closing
   [junit4]   2> 143435 INFO  (OverseerStateUpdate-98188836055023625-127.0.0.1:48656_solr-n_0000000000) [n:127.0.0.1:48656_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:48656_solr
   [junit4]   2> 143442 WARN  (zkCallback-709-thread-4-processing-n:127.0.0.1:42704_solr) [n:127.0.0.1:42704_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 143451 INFO  (jetty-closer-698-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@644a1e63{/solr,null,UNAVAILABLE}
   [junit4]   2> 144949 WARN  (zkCallback-714-thread-3-processing-n:127.0.0.1:48656_solr) [n:127.0.0.1:48656_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 144950 INFO  (jetty-closer-698-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@74f6720d{/solr,null,UNAVAILABLE}
   [junit4]   2> 144951 ERROR (SUITE-CloudSolrClientTest-seed#[7CAF511051E022C3]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 144951 INFO  (SUITE-CloudSolrClientTest-seed#[7CAF511051E022C3]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32894 32894
   [junit4]   2> 145062 INFO  (Thread-143) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:32894 32894
   [junit4]   2> 145063 WARN  (Thread-143) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/aliases.json
   [junit4]   2> 7 /solr/clusterprops.json
   [junit4]   2> 3 /solr/security.json
   [junit4]   2> 3 /solr/configs/conf
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/clusterstate.json
   [junit4]   2> 3 /solr/collections/foo/state.json
   [junit4]   2> 3 /solr/collections/localShardsTestColl/state.json
   [junit4]   2> 2 /solr/collections/foo/leader_elect/shard1/election/98188836055023625-core_node2-n_0000000000
   [junit4]   2> 2 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/collections/multicollection2/state.json
   [junit4]   2> 2 /solr/collections/nemesis/state.json
   [junit4]   2> 2 /solr/collections/multicollection1/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/live_nodes
   [junit4]   2> 7 /solr/collections
   [junit4]   2>
   [junit4]   2> 145063 INFO  (SUITE-CloudSolrClientTest-seed#[7CAF511051E022C3]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_7CAF511051E022C3-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=bg, timezone=Canada/Yukon
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=76704296,total=328204288
   [junit4]   2> NOTE: All tests run in this JVM: [JdbcTest, ModifiableSolrParamsTest, CubedRootEvaluatorTest, NoOpResponseParserTest, CompoundEvaluatorTest, TestBatchUpdate, StreamingTest, TestHash, JettyWebappTest, TestSuggesterResponse, TestJavaBinCodec, GraphExpressionTest, SolrParamTest, HyperbolicSineEvaluatorTest, SolrExampleBinaryTest, SolrExampleJettyTest, SolrExampleXMLTest, StreamExpressionParserTest, LessThanEvaluatorTest, PowerEvaluatorTest, ConcurrentUpdateSolrClientTest, FacetFieldTest, MergeIndexesEmbeddedTest, SolrQueryTest, TestXMLEscaping, HttpClientUtilTest, DocumentAnalysisResponseTest, SolrDocumentTest, SolrExceptionTest, NamedListTest, AnlysisResponseBaseTest, GetByIdTest, CloudSolrClientBuilderTest, CloudSolrClientCacheTest, CloudSolrClientMultiConstructorTest, CloudSolrClientTest]
   [junit4] Completed [118/130 (1!)] on J2 in 36.41s, 15 tests, 1 error <<< FAILURES!

[...truncated 42673 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 380 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/380/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not find collection:delete_data_dir

Stack Trace:
java.lang.AssertionError: Could not find collection:delete_data_dir
        at __randomizedtesting.SeedInfo.seed([489BEF0CF8B8FA57:C0CFD0D6564497AF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:159)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13326 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/init-core-data-001
   [junit4]   2> 3222312 WARN  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=39 numCloses=39
   [junit4]   2> 3222312 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 3222314 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3222314 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3222343 WARN  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3222347 WARN  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3222348 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3222358 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_38138_hdfs____z1sor/webapp
   [junit4]   2> 3222742 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38138
   [junit4]   2> 3222796 WARN  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3222797 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3222807 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36127_datanode____.emkfc6/webapp
   [junit4]   2> 3223203 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36127
   [junit4]   2> 3223235 WARN  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3223235 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3223244 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37004_datanode____.gq1brd/webapp
   [junit4]   2> 3223280 INFO  (IPC Server handler 3 on 33734) [    ] BlockStateChange BLOCK* processReport: from storage DS-bed96892-f730-4de0-9e87-235444e14304 node DatanodeRegistration(127.0.0.1:54618, datanodeUuid=149379b5-f216-4524-ace8-23d4c397156c, infoPort=46777, infoSecurePort=0, ipcPort=33306, storageInfo=lv=-56;cid=testClusterID;nsid=1308266784;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3223280 INFO  (IPC Server handler 3 on 33734) [    ] BlockStateChange BLOCK* processReport: from storage DS-0fa77e2e-4def-44b5-bb52-116deacf24e7 node DatanodeRegistration(127.0.0.1:54618, datanodeUuid=149379b5-f216-4524-ace8-23d4c397156c, infoPort=46777, infoSecurePort=0, ipcPort=33306, storageInfo=lv=-56;cid=testClusterID;nsid=1308266784;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3223646 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37004
   [junit4]   2> 3223719 INFO  (IPC Server handler 8 on 33734) [    ] BlockStateChange BLOCK* processReport: from storage DS-04f87660-5f76-4fc2-a879-7e3320423943 node DatanodeRegistration(127.0.0.1:52895, datanodeUuid=f1eb2e8d-2be7-4575-a293-a4bebea9b5c3, infoPort=47466, infoSecurePort=0, ipcPort=38978, storageInfo=lv=-56;cid=testClusterID;nsid=1308266784;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3223720 INFO  (IPC Server handler 8 on 33734) [    ] BlockStateChange BLOCK* processReport: from storage DS-352e0508-8457-42b9-a535-9caa6826af43 node DatanodeRegistration(127.0.0.1:52895, datanodeUuid=f1eb2e8d-2be7-4575-a293-a4bebea9b5c3, infoPort=47466, infoSecurePort=0, ipcPort=38978, storageInfo=lv=-56;cid=testClusterID;nsid=1308266784;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3223794 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3223794 INFO  (Thread-60068) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3223794 INFO  (Thread-60068) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3223795 ERROR (Thread-60068) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3223894 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ZkTestServer start zk server on port:53945
   [junit4]   2> 3223900 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3223901 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3223903 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3223903 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3223904 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3223905 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3223906 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3223906 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3223907 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3223908 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3223909 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3223983 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/control-001/cores/collection1
   [junit4]   2> 3223984 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3223985 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f24721a{/,null,AVAILABLE}
   [junit4]   2> 3223985 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7785397e{HTTP/1.1,[http/1.1]}{127.0.0.1:40334}
   [junit4]   2> 3223986 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server Started @3226386ms
   [junit4]   2> 3223986 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33734/hdfs__localhost_33734__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001_tempDir-002_control_data, hostContext=/, hostPort=40334, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/control-001/cores}
   [junit4]   2> 3223986 ERROR (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3223987 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3223987 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3223987 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3223987 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-24T13:00:34.222Z
   [junit4]   2> 3223988 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3223988 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/control-001/solr.xml
   [junit4]   2> 3223993 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3223994 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53945/solr
   [junit4]   2> 3224005 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:40334_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3224005 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:40334_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40334_
   [junit4]   2> 3224006 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:40334_    ] o.a.s.c.Overseer Overseer (id=98193193961390084-127.0.0.1:40334_-n_0000000000) starting
   [junit4]   2> 3224010 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:40334_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40334_
   [junit4]   2> 3224011 INFO  (zkCallback-4102-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3224052 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:40334_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/control-001/cores
   [junit4]   2> 3224053 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:40334_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3224053 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3224054 INFO  (OverseerStateUpdate-98193193961390084-127.0.0.1:40334_-n_0000000000) [n:127.0.0.1:40334_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3225067 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3225081 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3225199 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3225209 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3225209 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33734/solr_hdfs_home
   [junit4]   2> 3225209 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3225209 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3225209 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/control-001/cores/collection1], dataDir=[hdfs://localhost:33734/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3225209 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f4b18eb
   [junit4]   2> 3225210 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 3225215 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3225215 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3225215 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3225222 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3225223 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3225234 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3225239 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3225239 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3225239 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3225245 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3225245 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3225252 INFO  (IPC Server handler 1 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54618 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-04f87660-5f76-4fc2-a879-7e3320423943:NORMAL:127.0.0.1:52895|RBW], ReplicaUC[[DISK]DS-0fa77e2e-4def-44b5-bb52-116deacf24e7:NORMAL:127.0.0.1:54618|FINALIZED]]} size 0
   [junit4]   2> 3225253 INFO  (IPC Server handler 2 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52895 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-04f87660-5f76-4fc2-a879-7e3320423943:NORMAL:127.0.0.1:52895|RBW], ReplicaUC[[DISK]DS-0fa77e2e-4def-44b5-bb52-116deacf24e7:NORMAL:127.0.0.1:54618|FINALIZED]]} size 0
   [junit4]   2> 3225257 WARN  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3225283 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3225283 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3225283 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3225301 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3225301 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3225302 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=97.20703125, floorSegmentMB=1.705078125, forceMergeDeletesPctAllowed=11.659843892170086, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6190997824936103
   [junit4]   2> 3225304 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3be97c54[collection1] main]
   [junit4]   2> 3225305 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3225305 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3225305 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3225306 INFO  (searcherExecutor-8802-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3be97c54[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3225307 INFO  (coreLoadExecutor-8801-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571091104967688192
   [junit4]   2> 3225311 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3225311 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3225311 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40334/collection1/
   [junit4]   2> 3225312 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3225312 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40334/collection1/ has no replicas
   [junit4]   2> 3225312 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3225314 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40334/collection1/ shard1
   [junit4]   2> 3225464 INFO  (coreZkRegister-8794-thread-1-processing-n:127.0.0.1:40334_ x:collection1 c:control_collection) [n:127.0.0.1:40334_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3225561 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3225561 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53945/solr ready
   [junit4]   2> 3225562 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3225562 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 3225642 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-1-001/cores/collection1
   [junit4]   2> 3225643 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-1-001
   [junit4]   2> 3225643 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3225644 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2c96560e{/,null,AVAILABLE}
   [junit4]   2> 3225644 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74074c79{HTTP/1.1,[http/1.1]}{127.0.0.1:52331}
   [junit4]   2> 3225644 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server Started @3228044ms
   [junit4]   2> 3225645 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33734/hdfs__localhost_33734__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=52331, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-1-001/cores}
   [junit4]   2> 3225645 ERROR (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3225645 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3225646 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3225646 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3225646 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-24T13:00:35.881Z
   [junit4]   2> 3225647 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3225647 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-1-001/solr.xml
   [junit4]   2> 3225652 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3225653 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53945/solr
   [junit4]   2> 3225658 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3225659 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52331_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3225661 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52331_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52331_
   [junit4]   2> 3225661 INFO  (zkCallback-4106-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3225662 INFO  (zkCallback-4102-thread-1-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3225666 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3225698 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52331_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-1-001/cores
   [junit4]   2> 3225698 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52331_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3225698 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3225699 INFO  (OverseerStateUpdate-98193193961390084-127.0.0.1:40334_-n_0000000000) [n:127.0.0.1:40334_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3225801 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3226711 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3226725 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3226831 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3226840 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3226841 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33734/solr_hdfs_home
   [junit4]   2> 3226841 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3226841 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3226841 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:33734/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 3226841 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f4b18eb
   [junit4]   2> 3226841 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 3226847 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3226847 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3226847 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3226852 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3226853 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 3226865 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 3226870 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3226870 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3226870 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3226875 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3226875 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3226883 INFO  (IPC Server handler 9 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52895 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bed96892-f730-4de0-9e87-235444e14304:NORMAL:127.0.0.1:54618|RBW], ReplicaUC[[DISK]DS-352e0508-8457-42b9-a535-9caa6826af43:NORMAL:127.0.0.1:52895|FINALIZED]]} size 0
   [junit4]   2> 3226883 INFO  (IPC Server handler 0 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54618 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bed96892-f730-4de0-9e87-235444e14304:NORMAL:127.0.0.1:54618|RBW], ReplicaUC[[DISK]DS-352e0508-8457-42b9-a535-9caa6826af43:NORMAL:127.0.0.1:52895|FINALIZED]]} size 0
   [junit4]   2> 3226889 WARN  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3226917 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3226917 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3226917 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3226924 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3226924 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3226926 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=97.20703125, floorSegmentMB=1.705078125, forceMergeDeletesPctAllowed=11.659843892170086, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6190997824936103
   [junit4]   2> 3226932 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@a81f62a[collection1] main]
   [junit4]   2> 3226935 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3226936 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3226936 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3226937 INFO  (searcherExecutor-8813-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@a81f62a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3226938 INFO  (coreLoadExecutor-8812-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571091106677915648
   [junit4]   2> 3226943 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3226943 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3226943 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52331/collection1/
   [junit4]   2> 3226943 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3226943 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:52331/collection1/ has no replicas
   [junit4]   2> 3226943 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3226945 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52331/collection1/ shard1
   [junit4]   2> 3227047 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3227096 INFO  (coreZkRegister-8807-thread-1-processing-n:127.0.0.1:52331_ x:collection1 c:collection1) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3227199 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3227278 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-2-001/cores/collection1
   [junit4]   2> 3227278 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-2-001
   [junit4]   2> 3227279 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3227280 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@20494594{/,null,AVAILABLE}
   [junit4]   2> 3227280 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2102b238{HTTP/1.1,[http/1.1]}{127.0.0.1:56478}
   [junit4]   2> 3227280 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server Started @3229680ms
   [junit4]   2> 3227281 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33734/hdfs__localhost_33734__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=56478, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-2-001/cores}
   [junit4]   2> 3227281 ERROR (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3227281 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3227281 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3227281 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3227281 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-24T13:00:37.516Z
   [junit4]   2> 3227283 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3227283 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-2-001/solr.xml
   [junit4]   2> 3227288 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3227289 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53945/solr
   [junit4]   2> 3227294 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:56478_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3227295 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:56478_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3227297 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:56478_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56478_
   [junit4]   2> 3227297 INFO  (zkCallback-4102-thread-2-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3227297 INFO  (zkCallback-4106-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3227297 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3227302 INFO  (zkCallback-4118-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3227369 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:56478_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-2-001/cores
   [junit4]   2> 3227370 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:56478_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3227372 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3227374 INFO  (OverseerStateUpdate-98193193961390084-127.0.0.1:40334_-n_0000000000) [n:127.0.0.1:40334_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3227475 INFO  (zkCallback-4118-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3227475 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3228390 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3228417 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3228524 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3228535 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3228535 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33734/solr_hdfs_home
   [junit4]   2> 3228535 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3228536 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3228536 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:33734/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 3228536 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f4b18eb
   [junit4]   2> 3228536 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 3228541 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3228541 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3228541 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3228546 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3228547 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 3228557 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 3228562 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3228562 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3228562 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3228566 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3228567 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3228572 INFO  (IPC Server handler 8 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54618 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-352e0508-8457-42b9-a535-9caa6826af43:NORMAL:127.0.0.1:52895|RBW], ReplicaUC[[DISK]DS-0fa77e2e-4def-44b5-bb52-116deacf24e7:NORMAL:127.0.0.1:54618|FINALIZED]]} size 0
   [junit4]   2> 3228572 INFO  (IPC Server handler 9 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52895 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0fa77e2e-4def-44b5-bb52-116deacf24e7:NORMAL:127.0.0.1:54618|FINALIZED], ReplicaUC[[DISK]DS-04f87660-5f76-4fc2-a879-7e3320423943:NORMAL:127.0.0.1:52895|FINALIZED]]} size 0
   [junit4]   2> 3228577 WARN  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3228604 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3228604 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3228604 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3228612 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3228612 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3228614 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=97.20703125, floorSegmentMB=1.705078125, forceMergeDeletesPctAllowed=11.659843892170086, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6190997824936103
   [junit4]   2> 3228616 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3dc252fe[collection1] main]
   [junit4]   2> 3228617 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3228617 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3228617 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3228618 INFO  (searcherExecutor-8824-thread-1-processing-n:127.0.0.1:56478_ x:collection1 c:collection1) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3dc252fe[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3228619 INFO  (coreLoadExecutor-8823-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1571091108440571904
   [junit4]   2> 3228622 INFO  (coreZkRegister-8818-thread-1-processing-n:127.0.0.1:56478_ x:collection1 c:collection1) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 3228623 INFO  (updateExecutor-4115-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 3228623 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 3228623 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 3228623 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 3228624 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. HDFSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 3228624 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:52331/collection1/] and I am [http://127.0.0.1:56478/collection1/]
   [junit4]   2> 3228625 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:52331]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:56478_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 3228626 INFO  (qtp1850830898-77351) [n:127.0.0.1:52331_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 3228626 INFO  (qtp1850830898-77351) [n:127.0.0.1:52331_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 3228626 INFO  (qtp1850830898-77351) [n:127.0.0.1:52331_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:56478_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:56478","node_name":"127.0.0.1:56478_","state":"down"}
   [junit4]   2> 3228727 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3228727 INFO  (zkCallback-4118-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3228946 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-3-001/cores/collection1
   [junit4]   2> 3228946 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-3-001
   [junit4]   2> 3228947 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3228948 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3fde9863{/,null,AVAILABLE}
   [junit4]   2> 3228948 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@18d5216c{HTTP/1.1,[http/1.1]}{127.0.0.1:52402}
   [junit4]   2> 3228949 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.e.j.s.Server Started @3231348ms
   [junit4]   2> 3228949 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:33734/hdfs__localhost_33734__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001_tempDir-002_jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=52402, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-3-001/cores}
   [junit4]   2> 3228949 ERROR (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3228949 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.7.0
   [junit4]   2> 3228949 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3228949 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3228950 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-06-24T13:00:39.185Z
   [junit4]   2> 3228951 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3228951 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-3-001/solr.xml
   [junit4]   2> 3228956 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3228957 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53945/solr
   [junit4]   2> 3228963 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52402_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3228964 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52402_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3228965 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52402_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52402_
   [junit4]   2> 3228966 INFO  (zkCallback-4118-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3228966 INFO  (zkCallback-4106-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3228966 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3228966 INFO  (zkCallback-4102-thread-2-processing-n:127.0.0.1:40334_) [n:127.0.0.1:40334_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3228966 INFO  (zkCallback-4125-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3229017 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52402_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-3-001/cores
   [junit4]   2> 3229017 INFO  (TEST-StressHdfsTest.test-seed#[489BEF0CF8B8FA57]) [n:127.0.0.1:52402_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3229017 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3229018 INFO  (OverseerStateUpdate-98193193961390084-127.0.0.1:40334_-n_0000000000) [n:127.0.0.1:40334_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3229120 INFO  (zkCallback-4112-thread-1-processing-n:127.0.0.1:52331_) [n:127.0.0.1:52331_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3229120 INFO  (zkCallback-4118-thread-1-processing-n:127.0.0.1:56478_) [n:127.0.0.1:56478_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3229120 INFO  (zkCallback-4125-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3229626 INFO  (qtp1850830898-77351) [n:127.0.0.1:52331_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:56478_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","dataDir":"hdfs://localhost:33734/solr_hdfs_home/collection1/core_node2/data/","base_url":"http://127.0.0.1:56478","node_name":"127.0.0.1:56478_","state":"recovering","ulogDir":"hdfs://localhost:33734/solr_hdfs_home/collection1/core_node2/data/tlog"}
   [junit4]   2> 3229626 INFO  (qtp1850830898-77351) [n:127.0.0.1:52331_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 3229626 INFO  (qtp1850830898-77351) [n:127.0.0.1:52331_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:56478_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
   [junit4]   2> 3230029 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 3230043 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3230127 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:52331/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 3230127 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:56478 START replicas=[http://127.0.0.1:52331/collection1/] nUpdates=100
   [junit4]   2> 3230129 INFO  (qtp1850830898-77355) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3230129 INFO  (qtp1850830898-77355) [n:127.0.0.1:52331_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3230130 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:1.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3230130 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 3230130 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3230130 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 3230130 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3230130 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 3230131 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 3230131 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 3230131 INFO  (recoveryExecutor-4116-thread-1-processing-n:127.0.0.1:56478_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:56478_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 3230156 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3230165 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3230166 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:33734/solr_hdfs_home
   [junit4]   2> 3230166 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3230166 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3230166 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001/shard-3-001/cores/collection1], dataDir=[hdfs://localhost:33734/solr_hdfs_home/collection1/core_node3/data/]
   [junit4]   2> 3230166 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f4b18eb
   [junit4]   2> 3230166 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node3/data/snapshot_metadata
   [junit4]   2> 3230172 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3230172 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3230172 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3230177 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3230178 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node3/data
   [junit4]   2> 3230190 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:33734/solr_hdfs_home/collection1/core_node3/data/index
   [junit4]   2> 3230195 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3230195 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3230195 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3230201 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3230201 INFO  (coreLoadExecutor-8834-thread-1-processing-n:127.0.0.1:52402_) [n:127.0.0.1:52402_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3230206 INFO  (IPC Server handler 1 on 33734) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54618

[...truncated too long message...]

rics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:203)
   [junit4]   2> at org.apache.solr.cloud.hdfs.StressHdfsTest.teardownClass(StressHdfsTest.java:77)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 53 more
   [junit4]   2> 3348628 INFO  (SUITE-StressHdfsTest-seed#[489BEF0CF8B8FA57]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_489BEF0CF8B8FA57-001
   [junit4]   2> Jun 24, 2017 1:02:38 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=25955, maxDocsPerChunk=676, blockSize=2), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=25955, blockSize=2)), sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=mt, timezone=Pacific/Yap
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=169745456,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedSuggestComponentTest, SimpleCollectionCreateDeleteTest, TestSolrXml, SimpleFacetsTest, TestConfigSetsAPIZkFailure, ActionThrottleTest, TestSurroundQueryParser, TestConfigSetImmutable, TestSimpleQParserPlugin, TestDynamicFieldCollectionResource, TestSolrCoreProperties, UUIDUpdateProcessorFallbackTest, TestHdfsBackupRestoreCore, TestInitQParser, JsonLoaderTest, TestCloudInspectUtil, CheckHdfsIndexTest, ResponseLogComponentTest, DirectSolrSpellCheckerTest, NumericFieldsTest, TestImplicitCoreProperties, RecoveryZkTest, PolyFieldTest, HttpSolrCallGetCoreTest, SolrJmxReporterTest, TermVectorComponentDistributedTest, TestMinMaxOnMultiValuedField, TestPKIAuthenticationPlugin, HdfsRestartWhileUpdatingTest, TestSort, SolrCoreMetricManagerTest, TestSolrCloudWithKerberosAlt, TestDocumentBuilder, TestZkChroot, BooleanFieldTest, CollectionTooManyReplicasTest, TestSchemaVersionResource, TestSolrDeletionPolicy2, TestDistribDocBasedVersion, OverriddenZkACLAndCredentialsProvidersTest, DOMUtilTest, MetricUtilsTest, TimeZoneUtilsTest, TestFoldingMultitermQuery, TestTolerantUpdateProcessorRandomCloud, BadIndexSchemaTest, TestLMJelinekMercerSimilarityFactory, TestPayloadCheckQParserPlugin, TestFieldResource, DeleteLastCustomShardedReplicaTest, TestMultiWordSynonyms, ClassificationUpdateProcessorFactoryTest, TestIndexSearcher, SuggesterTest, CdcrReplicationDistributedZkTest, MigrateRouteKeyTest, RegexBoostProcessorTest, XmlUpdateRequestHandlerTest, TestRecovery, SolrCoreCheckLockOnStartupTest, CreateCollectionCleanupTest, DirectoryFactoryTest, TestExportWriter, SolrInfoMBeanTest, TestXmlQParserPlugin, BlockJoinFacetSimpleTest, HdfsChaosMonkeyNothingIsSafeTest, OverseerRolesTest, TestSolrCoreParser, TestAddFieldRealTimeGet, TestQueryTypes, V2ApiIntegrationTest, TestFieldCacheSort, TestRawTransformer, TestExceedMaxTermLength, TestFastLRUCache, AnalysisErrorHandlingTest, RulesTest, TestFieldCacheWithThreads, SolrTestCaseJ4Test, HLLSerializationTest, TestUseDocValuesAsStored, CopyFieldTest, TestCloudPseudoReturnFields, FileUtilsTest, TestFileDictionaryLookup, BitVectorTest, TestConfigsApi, TestRequestStatusCollectionAPI, TestMiniSolrCloudCluster, ConcurrentDeleteAndCreateCollectionTest, UnloadDistributedZkTest, CloneFieldUpdateProcessorFactoryTest, TestSolrQueryParserResource, LeaderElectionTest, TestNestedDocsSort, AnalyticsQueryTest, TestSolrCLIRunExample, HardAutoCommitTest, TestDFRSimilarityFactory, VMParamsZkACLAndCredentialsProvidersTest, TestRecoveryHdfs, MoreLikeThisHandlerTest, BJQParserTest, TestEmbeddedSolrServerSchemaAPI, DistributedVersionInfoTest, CdcrVersionReplicationTest, PeerSyncWithIndexFingerprintCachingTest, DistributedMLTComponentTest, TestUnifiedSolrHighlighter, DistributedExpandComponentTest, HttpPartitionTest, DirectUpdateHandlerTest, ParsingFieldUpdateProcessorsTest, TestReload, BasicDistributedZk2Test, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, ZkSolrClientTest, ZkCLITest, TestDistributedSearch, AutoCommitTest, TestHashPartitioner, DistributedSpellCheckComponentTest, ZkControllerTest, TestStressReorder, TestJoin, TestRangeQuery, TestSolr4Spatial, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, PeerSyncTest, BasicFunctionalityTest, TestLazyCores, TestBadConfig, HighlighterTest, SoftAutoCommitTest, TestPseudoReturnFields, QueryEqualityTest, TestRemoteStreaming, XsltUpdateRequestHandlerTest, CacheHeaderTest, TestQueryUtils, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, RequestHandlersTest, RequiredFieldsTest, TestSolrQueryParser, SolrPluginUtilsTest, TestJmxIntegration, MBeansHandlerTest, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, TestComponentsName, SearchHandlerTest, SOLR749Test, UpdateParamsTest, AlternateDirectoryTest, TestQuerySenderNoQuery, BadComponentTest, MultiTermTest, TestDocSet, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, EchoParamsTest, TestSweetSpotSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, AliasIntegrationTest, CdcrReplicationHandlerTest, DistribCursorPagingTest, LeaderElectionContextKeyTest, LeaderInitiatedRecoveryOnCommitTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, PeerSyncReplicationTest, RemoteQueryErrorTest, ReplaceNodeTest, ReplicationFactorTest, RestartWhileUpdatingTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestCloudJSONFacetJoinDomain, TestSizeLimitedDistributedMap, TestStressInPlaceUpdates, HdfsRecoverLeaseTest, HdfsRecoveryZkTest, HdfsUnloadDistributedZkTest, StressHdfsTest]
   [junit4] Completed [560/713 (1!)] on J2 in 136.28s, 1 test, 1 failure <<< FAILURES!

[...truncated 43917 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]