[JENKINS] Lucene-Solr-Tests-7.0 - Build # 133 - Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.0 - Build # 133 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.0/133/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestStressInPlaceUpdates.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=20282, name=WRITER2, state=RUNNABLE, group=TGRP-TestStressInPlaceUpdates]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20282, name=WRITER2, state=RUNNABLE, group=TGRP-TestStressInPlaceUpdates]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:49112/collection1: Cannot talk to ZooKeeper - Updates are disabled.
        at __randomizedtesting.SeedInfo.seed([EF2A1348D287BB44]:0)
        at org.apache.solr.cloud.TestStressInPlaceUpdates$1.run(TestStressInPlaceUpdates.java:306)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:49112/collection1: Cannot talk to ZooKeeper - Updates are disabled.
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.TestStressInPlaceUpdates$1.run(TestStressInPlaceUpdates.java:169)




Build Log:
[...truncated 12652 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressInPlaceUpdates
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/init-core-data-001
   [junit4]   2> 3577486 WARN  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 3577486 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3577487 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3577487 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 3577490 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 3577490 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 3577595 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3577716 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
   [junit4]   2> 3577966 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 3578162 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3578191 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3578191 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3578197 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 3578242 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3578262 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 3578265 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 3578265 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 3578266 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3578266 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3578270 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/init-core-data-001/]
   [junit4]   2> 3578272 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=12.3115234375, floorSegmentMB=0.3662109375, forceMergeDeletesPctAllowed=28.308600613926327, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1342914256719907
   [junit4]   2> 3578277 WARN  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3578351 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3578351 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3578352 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3578352 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3578361 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=18.6806640625, floorSegmentMB=0.4033203125, forceMergeDeletesPctAllowed=18.106691962156436, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3578372 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c6db897[collection1] main]
   [junit4]   2> 3578373 WARN  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 3578373 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3578373 INFO  (coreLoadExecutor-7214-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578362947169681408
   [junit4]   2> 3578374 INFO  (searcherExecutor-7215-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7c6db897[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3578376 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 3578378 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3578390 INFO  (Thread-5990) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3578390 INFO  (Thread-5990) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3578512 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ZkTestServer start zk server on port:59367
   [junit4]   2> 3578563 ERROR (Thread-5990) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3579010 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3579011 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
   [junit4]   2> 3579011 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3579012 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3579014 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3579015 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3579015 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3579016 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3579016 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3579017 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3579017 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3579032 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 3579272 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3579427 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f5939a3{/,null,AVAILABLE}
   [junit4]   2> 3579427 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@77eb7b59{SSL,[ssl, http/1.1]}{127.0.0.1:43812}
   [junit4]   2> 3579427 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server Started @3592412ms
   [junit4]   2> 3579427 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/tempDir-001/control/data, hostContext=/, hostPort=43812, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/control-001/cores}
   [junit4]   2> 3579428 ERROR (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3579428 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3579428 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3579428 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3579428 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T19:23:25.738Z
   [junit4]   2> 3579461 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3579461 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/control-001/solr.xml
   [junit4]   2> 3579466 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3579478 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59367/solr
   [junit4]   2> 3580038 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3580038 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43812_
   [junit4]   2> 3580039 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.c.Overseer Overseer (id=98647684199940101-127.0.0.1:43812_-n_0000000000) starting
   [junit4]   2> 3580082 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43812_
   [junit4]   2> 3580142 INFO  (OverseerStateUpdate-98647684199940101-127.0.0.1:43812_-n_0000000000) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3582016 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3582133 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3582133 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3582143 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43812_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/control-001/cores
   [junit4]   2> 3582210 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3582211 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59367/solr ready
   [junit4]   2> 3582266 INFO  (qtp195673936-20144) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:43812_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3582267 INFO  (OverseerThreadFactory-7225-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 3582471 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3582471 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3582573 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3586128 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3586140 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=inplace-updates
   [junit4]   2> 3586164 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 3586164 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3586165 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3586165 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3586165 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 3586182 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=12.3115234375, floorSegmentMB=0.3662109375, forceMergeDeletesPctAllowed=28.308600613926327, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1342914256719907
   [junit4]   2> 3586186 WARN  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3586507 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3586507 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3586508 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3586508 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3586526 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=18.6806640625, floorSegmentMB=0.4033203125, forceMergeDeletesPctAllowed=18.106691962156436, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3586527 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c22b676[control_collection_shard1_replica_n1] main]
   [junit4]   2> 3586528 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3586528 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3586528 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3586529 INFO  (searcherExecutor-7228-thread-1-processing-n:127.0.0.1:43812_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@3c22b676[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3586529 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578362955721867264
   [junit4]   2> 3586536 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3586537 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3586537 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3586537 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:43812/control_collection_shard1_replica_n1/
   [junit4]   2> 3586537 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3586537 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:43812/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3586537 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3586540 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:43812/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 3586711 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3586760 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3586761 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=4290
   [junit4]   2> 3586763 INFO  (qtp195673936-20144) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3586864 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3587766 INFO  (qtp195673936-20144) [n:127.0.0.1:43812_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:43812_&wt=javabin&version=2} status=0 QTime=5500
   [junit4]   2> 3587926 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3587927 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59367/solr ready
   [junit4]   2> 3587927 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3587934 INFO  (qtp195673936-20137) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3587935 INFO  (OverseerCollectionConfigSetProcessor-98647684199940101-127.0.0.1:43812_-n_0000000000) [n:127.0.0.1:43812_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3587935 INFO  (OverseerThreadFactory-7225-thread-2-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 3587936 WARN  (OverseerThreadFactory-7225-thread-2-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 3588227 INFO  (qtp195673936-20137) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3588228 INFO  (qtp195673936-20137) [n:127.0.0.1:43812_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=294
   [junit4]   2> 3588751 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-1-001 of type NRT
   [junit4]   2> 3588771 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3588798 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@453fee51{/,null,AVAILABLE}
   [junit4]   2> 3588799 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1950a521{SSL,[ssl, http/1.1]}{127.0.0.1:51371}
   [junit4]   2> 3588799 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server Started @3601783ms
   [junit4]   2> 3588799 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=51371, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-1-001/cores}
   [junit4]   2> 3588799 ERROR (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3588799 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3588799 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3588799 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3588800 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T19:23:35.110Z
   [junit4]   2> 3588830 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3588830 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-1-001/solr.xml
   [junit4]   2> 3588834 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3588836 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59367/solr
   [junit4]   2> 3588841 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3588842 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3588847 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51371_
   [junit4]   2> 3588848 INFO  (zkCallback-3015-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3588854 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3588873 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3589221 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3589246 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3589246 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3589247 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:51371_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-1-001/cores
   [junit4]   2> 3589293 INFO  (qtp195673936-20141) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:51371_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3589301 INFO  (OverseerCollectionConfigSetProcessor-98647684199940101-127.0.0.1:43812_-n_0000000000) [n:127.0.0.1:43812_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3589306 INFO  (OverseerThreadFactory-7225-thread-3-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:51371_ for creating new replica
   [junit4]   2> 3589348 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3589348 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3589452 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3590361 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3591007 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema [collection1_shard1_replica_n0] Schema name=inplace-updates
   [junit4]   2> 3591059 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 3591059 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 3591059 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n0' (registry 'solr.core.collection1.shard1.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3591059 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3591059 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [[collection1_shard1_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-1-001/cores/collection1_shard1_replica_n0], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-1-001/cores/collection1_shard1_replica_n0/data/]
   [junit4]   2> 3591068 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=12.3115234375, floorSegmentMB=0.3662109375, forceMergeDeletesPctAllowed=28.308600613926327, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1342914256719907
   [junit4]   2> 3591070 WARN  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3592204 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3592204 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3592205 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3592205 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3592209 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=18.6806640625, floorSegmentMB=0.4033203125, forceMergeDeletesPctAllowed=18.106691962156436, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3592210 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@3723fed2[collection1_shard1_replica_n0] main]
   [junit4]   2> 3592211 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3592212 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3592213 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3592218 INFO  (searcherExecutor-7239-thread-1-processing-n:127.0.0.1:51371_ x:collection1_shard1_replica_n0 s:shard1 c:collection1) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [collection1_shard1_replica_n0] Registered new searcher Searcher@3723fed2[collection1_shard1_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3592218 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578362961687216128
   [junit4]   2> 3592227 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3592227 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3592227 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3592227 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:51371/collection1_shard1_replica_n0/
   [junit4]   2> 3592227 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3592227 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:51371/collection1_shard1_replica_n0/ has no replicas
   [junit4]   2> 3592227 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3592230 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:51371/collection1_shard1_replica_n0/ shard1
   [junit4]   2> 3592331 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3592411 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3592413 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3065
   [junit4]   2> 3592448 INFO  (qtp195673936-20141) [n:127.0.0.1:43812_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:51371_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=3155
   [junit4]   2> 3592554 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3593316 INFO  (OverseerCollectionConfigSetProcessor-98647684199940101-127.0.0.1:43812_-n_0000000000) [n:127.0.0.1:43812_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3593590 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-2-001 of type NRT
   [junit4]   2> 3593591 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3594522 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55eb99{/,null,AVAILABLE}
   [junit4]   2> 3594522 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@e912a23{SSL,[ssl, http/1.1]}{127.0.0.1:49112}
   [junit4]   2> 3594522 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server Started @3607507ms
   [junit4]   2> 3594522 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=49112, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-2-001/cores}
   [junit4]   2> 3594522 ERROR (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3594550 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3594550 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3594550 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3594550 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T19:23:40.860Z
   [junit4]   2> 3594652 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3594652 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-2-001/solr.xml
   [junit4]   2> 3594656 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3594774 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59367/solr
   [junit4]   2> 3594812 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3594815 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3594816 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49112_
   [junit4]   2> 3594817 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3594823 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3594823 INFO  (zkCallback-3015-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3594848 INFO  (zkCallback-3026-thread-2-processing-n:127.0.0.1:49112_) [n:127.0.0.1:49112_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3595171 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3595198 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3595198 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3595199 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:49112_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-2-001/cores
   [junit4]   2> 3595286 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:49112_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3595288 INFO  (OverseerThreadFactory-7225-thread-4-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:49112_ for creating new replica
   [junit4]   2> 3595377 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3596137 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3596141 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3596184 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3596204 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=inplace-updates
   [junit4]   2> 3596208 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 3596208 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3596208 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3596208 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3596208 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-2-001/cores/collection1_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-2-001/cores/collection1_shard1_replica_n1/data/]
   [junit4]   2> 3596210 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=12.3115234375, floorSegmentMB=0.3662109375, forceMergeDeletesPctAllowed=28.308600613926327, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1342914256719907
   [junit4]   2> 3596212 WARN  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3596319 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3596319 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3597110 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3597110 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3597112 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=18.6806640625, floorSegmentMB=0.4033203125, forceMergeDeletesPctAllowed=18.106691962156436, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3597112 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@78f0283c[collection1_shard1_replica_n1] main]
   [junit4]   2> 3597113 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3597113 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3597114 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3597114 INFO  (searcherExecutor-7250-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@78f0283c[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3597114 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578362966821044224
   [junit4]   2> 3597118 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ZkController Core needs to recover:collection1_shard1_replica_n1
   [junit4]   2> 3597126 INFO  (qtp1978219411-20214) [n:127.0.0.1:49112_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1749
   [junit4]   2> 3597127 INFO  (qtp195673936-20142) [n:127.0.0.1:43812_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:49112_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1841
   [junit4]   2> 3597129 INFO  (updateExecutor-3023-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 3597151 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 3597152 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 3597152 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1_shard1_replica_n1]
   [junit4]   2> 3597152 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 3597152 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1_shard1_replica_n1] as recovering, leader is [https://127.0.0.1:51371/collection1_shard1_replica_n0/] and I am [https://127.0.0.1:49112/collection1_shard1_replica_n1/]
   [junit4]   2> 3597172 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:51371]; [WaitForState: action=PREPRECOVERY&core=collection1_shard1_replica_n0&nodeName=127.0.0.1:49112_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 3597190 INFO  (qtp1634276917-20180) [n:127.0.0.1:51371_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, maxTime: 183 s
   [junit4]   2> 3597190 INFO  (qtp1634276917-20180) [n:127.0.0.1:51371_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1_shard1_replica_n0, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:49112_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1_shard1_replica_n1","base_url":"https://127.0.0.1:49112","node_name":"127.0.0.1:49112_","state":"down","type":"NRT"}
   [junit4]   2> 3597254 INFO  (zkCallback-3026-thread-2-processing-n:127.0.0.1:49112_) [n:127.0.0.1:49112_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3597254 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3597287 INFO  (OverseerCollectionConfigSetProcessor-98647684199940101-127.0.0.1:43812_-n_0000000000) [n:127.0.0.1:43812_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3597657 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-3-001 of type NRT
   [junit4]   2> 3597714 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3597826 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28cdd04{/,null,AVAILABLE}
   [junit4]   2> 3597826 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@ec22be1{SSL,[ssl, http/1.1]}{127.0.0.1:43769}
   [junit4]   2> 3598061 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.e.j.s.Server Started @3611046ms
   [junit4]   2> 3598061 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=43769, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-3-001/cores}
   [junit4]   2> 3598062 ERROR (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3598068 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3598068 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3598068 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3598068 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T19:23:44.378Z
   [junit4]   2> 3598191 INFO  (qtp1634276917-20180) [n:127.0.0.1:51371_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1_shard1_replica_n0, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:49112_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1_shard1_replica_n1","base_url":"https://127.0.0.1:49112","node_name":"127.0.0.1:49112_","state":"recovering","type":"NRT"}
   [junit4]   2> 3598191 INFO  (qtp1634276917-20180) [n:127.0.0.1:51371_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 3598191 INFO  (qtp1634276917-20180) [n:127.0.0.1:51371_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:49112_&onlyIfLeaderActive=true&core=collection1_shard1_replica_n0&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
   [junit4]   2> 3598236 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3598236 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-3-001/solr.xml
   [junit4]   2> 3598241 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3598270 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59367/solr
   [junit4]   2> 3598543 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3598543 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3598545 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43769_
   [junit4]   2> 3598546 INFO  (zkCallback-3015-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3598552 INFO  (zkCallback-3008-thread-1-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3598552 INFO  (zkCallback-3026-thread-2-processing-n:127.0.0.1:49112_) [n:127.0.0.1:49112_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3598552 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3598602 INFO  (zkCallback-3033-thread-2-processing-n:127.0.0.1:43769_) [n:127.0.0.1:43769_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 3598810 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:51371/collection1_shard1_replica_n0/] - recoveringAfterStartup=[true]
   [junit4]   2> 3598810 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_n1 url=https://127.0.0.1:49112 START replicas=[https://127.0.0.1:51371/collection1_shard1_replica_n0/] nUpdates=100
   [junit4]   2> 3598836 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n0] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3598836 INFO  (qtp1634276917-20183) [n:127.0.0.1:51371_ c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica_n0] o.a.s.c.S.Request [collection1_shard1_replica_n0]  webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 3598837 INFO  (recoveryExecutor-3024-thread-1-processing-n:127.0.0.1:49112_ x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:49112_ c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 3598928 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3598939 INFO  (zkCallback-3026-thread-2-processing-n:127.0.0.1:49112_) [n:127.0.0.1:49112_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3598939 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3598972 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3598972 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3598973 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [n:127.0.0.1:43769_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-3-001/cores
   [junit4]   2> 3599241 INFO  (qtp195673936-20144) [n:127.0.0.1:43812_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43769_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3599281 INFO  (OverseerThreadFactory-7225-thread-5-processing-n:127.0.0.1:43812_) [n:127.0.0.1:43812_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:43769_ for creating new replica
   [junit4]   2> 3599451 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3599451 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3599560 INFO  (zkCallback-3026-thread-2-processing-n:127.0.0.1:49112_) [n:127.0.0.1:49112_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3599560 INFO  (zkCallback-3033-thread-2-processing-n:127.0.0.1:43769_) [n:127.0.0.1:43769_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3599560 INFO  (zkCallback-3020-thread-2-processing-n:127.0.0.1:51371_) [n:127.0.0.1:51371_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 3600809 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3600867 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=inplace-updates
   [junit4]   2> 3600890 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 3600890 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 3600891 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6533d3d3
   [junit4]   2> 3600891 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3600891 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-3-001/cores/collection1_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressInPlaceUpdates_EF2A1348D287BB44-001/shard-3-001/cores/collection1_shard1_replica_n2/data/]
   [junit4]   2> 3601279 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=12.3115234375, floorSegmentMB=0.3662109375, forceMergeDeletesPctAllowed=28.308600613926327, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1342914256719907
   [junit4]   2> 3601283 WARN  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3601560 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3601560 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3601561 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3601561 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3601574 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=18.6806640625, floorSegmentMB=0.4033203125, forceMergeDeletesPctAllowed=18.106691962156436, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3601574 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@70db40eb[collection1_shard1_replica_n2] main]
   [junit4]   2> 3601575 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3601575 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3601576 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3601577 INFO  (searcherExecutor-7261-thread-1-processing-n:127.0.0.1:43769_ x:collection1_shard1_replica_n2 s:shard1 c:collection1) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [collection1_shard1_replica_n2] Registered new searcher Searcher@70db40eb[collection1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3601577 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578362971500838912
   [junit4]   2> 3601581 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ZkController Core needs to recover:collection1_shard1_replica_n2
   [junit4]   2> 3601590 INFO  (qtp2010131695-20247) [n:127.0.0.1:43769_ c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2139
   [junit4]   2> 3601591 INFO  (qtp195673936-20144) [n:127.0.0.1:43812_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:43769_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=2350
   [junit4]   2> 3601593 INFO  (updateExecutor-3030-thread-1-processing-n:127.0.0.1:43769_ x:collection1_shard1_replica_n2 s:shard1 c:collection1) [n:127.0.0.1:43769_ c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n2] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 3601605 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.SolrTestCaseJ4 ###Starting stressTest
   [junit4]   2> 3601605 INFO  (TEST-TestStressInPlaceUpdates.stressTest-seed#[EF2A1348D287BB44]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   1> replica:core_node1 rstate:active live:true
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node3 rstate:down live:true
   [junit4]   2> 3601704 INFO  (recoveryExecutor-3031-thread-1-processing-n:127.0.0.1:43769_ x:collection1_shard1_replica_n2 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:43769_ c:collection1 s:shard1 r:core_

[...truncated too long message...]

izedtesting.SeedInfo.seed([EF2A1348D287BB44]:0)
   [junit4]    > at org.apache.solr.cloud.TestStressInPlaceUpdates$1.run(TestStressInPlaceUpdates.java:306)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:43769/collection1: Cannot talk to ZooKeeper - Updates are disabled.
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
   [junit4]    > at org.apache.solr.cloud.TestStressInPlaceUpdates$1.run(TestStressInPlaceUpdates.java:173)
   [junit4]   2> 3657479 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 3657479 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=545216740
   [junit4]   2> 3657479 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3657479 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1b209177: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2f9aca30
   [junit4]   2> 3657480 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3657480 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e8fec16: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2201ee9d
   [junit4]   2> 3657481 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3657481 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e9becf1: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@513c71d3
   [junit4]   2> 3657481 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3657502 INFO  (coreCloseExecutor-7280-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4ac29a60
   [junit4]   2> 3657502 INFO  (coreCloseExecutor-7280-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1254267488
   [junit4]   2> 3657502 INFO  (coreCloseExecutor-7280-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@178ae599: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@40362345
   [junit4]   2> 3657590 INFO  (SUITE-TestStressInPlaceUpdates-seed#[EF2A1348D287BB44]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {title_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=Lucene50(blocksize=128), id_field_copy_that_does_not_support_in_place_update_s=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{val1_i_dvo=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Memory), val2_l_dvo=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Direct), inplace_updatable_int_with_default=DocValuesFormat(name=Memory), inplace_updatable_float_with_default=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=151, maxMBSortInHeap=5.184683205173828, sim=RandomSimilarity(queryNorm=true): {}, locale=es-UY, timezone=Libya
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=259904992,total=527433728
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastLRUCache, TestCloudPivotFacet, TestDeleteCollectionOnDownNodes, TestReqParamsAPI, CustomCollectionTest, TestCoreDiscovery, SampleTest, TestHashQParserPlugin, HighlighterTest, TestFieldCacheReopen, QueryParsingTest, TestSolrCloudWithHadoopAuthPlugin, CheckHdfsIndexTest, LeaderElectionContextKeyTest, TestSQLHandlerNonCloud, TestGeoJSONResponseWriter, TestTrieFacet, InfoHandlerTest, StatelessScriptUpdateProcessorFactoryTest, TestPostingsSolrHighlighter, LukeRequestHandlerTest, TestHttpShardHandlerFactory, MissingSegmentRecoveryTest, TestStressVersions, HdfsRecoveryZkTest, TestAnalyzeInfixSuggestions, TestCoreContainer, CdcrVersionReplicationTest, RequestLoggingTest, SpatialFilterTest, TestJsonFacetRefinement, SystemInfoHandlerTest, TestDistributedGrouping, CdcrReplicationDistributedZkTest, BasicAuthIntegrationTest, TestFieldSortValues, DocValuesNotIndexedTest, SchemaVersionSpecificBehaviorTest, TestFieldCacheVsDocValues, DeleteStatusTest, ClusterStateUpdateTest, DOMUtilTest, TestOrdValues, TestXIncludeConfig, BadComponentTest, TestShardHandlerFactory, TestBinaryResponseWriter, TestRandomRequestDistribution, CollectionsAPIDistributedZkTest, TestPushWriter, TestSchemaResource, DistribJoinFromCollectionTest, TestRTGBase, OverseerModifyCollectionTest, SliceStateTest, TlogReplayBufferedWhileIndexingTest, TestCursorMarkWithoutUniqueKey, DistributedMLTComponentTest, MetricUtilsTest, BasicFunctionalityTest, PrimUtilsTest, TestCloudDeleteByQuery, CollectionsAPIAsyncDistributedZkTest, TestDocBasedVersionConstraints, SolrMetricsIntegrationTest, TestDynamicLoading, GraphQueryTest, DistanceFunctionTest, MoreLikeThisHandlerTest, ZkControllerTest, RestartWhileUpdatingTest, TestSmileRequest, DocValuesTest, TestReloadDeadlock, TestAtomicUpdateErrorCases, TestFiltering, BlockJoinFacetDistribTest, TestMaxScoreQueryParser, OutOfBoxZkACLAndCredentialsProvidersTest, ShardRoutingCustomTest, TestSolrCLIRunExample, TestSubQueryTransformerCrossCore, TestSearchPerf, TestSolrIndexConfig, TestAuthorizationFramework, ShardRoutingTest, ClassificationUpdateProcessorTest, TestScoreJoinQPNoScore, TestApiFramework, AsyncCallRequestStatusResponseTest, TestPerFieldSimilarity, TestPullReplica, TestSystemIdResolver, ShufflingReplicaListTransformerTest, TestCoreAdminApis, ActionThrottleTest, TestSolrCloudWithDelegationTokens, DistributedIntervalFacetingTest, ZkSolrClientTest, DocValuesMultiTest, TestStressRecovery, HdfsLockFactoryTest, TestSolrJ, TestPayloadScoreQParserPlugin, SolrCloudReportersTest, RemoteQueryErrorTest, FileBasedSpellCheckerTest, HdfsThreadLeakTest, HdfsDirectoryFactoryTest, TestSolrQueryResponse, SortSpecParsingTest, TermVectorComponentTest, TestConfigSets, TestEmbeddedSolrServerSchemaAPI, SoftAutoCommitTest, TestPartialUpdateDeduplication, DisMaxRequestHandlerTest, TestValueSourceCache, DateRangeFieldTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, ZkCLITest, TestDistributedSearch, TermVectorComponentDistributedTest, TestRealTimeGet, TestJoin, TestRangeQuery, TestGroupingSearch, PeerSyncTest, AnalysisAfterCoreReloadTest, SuggesterFSTTest, CoreAdminHandlerTest, PolyFieldTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, TestRemoteStreaming, CacheHeaderTest, TestSurroundQueryParser, DirectSolrSpellCheckerTest, TestOmitPositions, PathHierarchyTokenizerFactoryTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, IndexSchemaTest, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, TestPHPSerializedResponseWriter, SearchHandlerTest, SOLR749Test, UpdateParamsTest, AlternateDirectoryTest, TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, TestDocSet, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TestCharFilters, TestCodecSupport, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, AliasIntegrationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CollectionReloadTest, ConfigSetsAPITest, ConnectionManagerTest, DeleteNodeTest, DeleteReplicaTest, DistributedVersionInfoTest, LeaderFailureAfterFreshStartTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, MoveReplicaTest, SolrCloudExampleTest, TestExclusionRuleCollectionAccess, TestLeaderElectionZkExpiry, TestMiniSolrCloudClusterSSL, TestRequestForwarding, TestSegmentSorting, TestSolrCloudWithSecureImpersonation, TestStressInPlaceUpdates]
   [junit4] Completed [544/729 (1!)] on J1 in 81.54s, 1 test, 1 error <<< FAILURES!

[...truncated 44039 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.0 - Build # 134 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.0/134/

5 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([BA976DB8DEF25536:32C35262700E38CE]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1114)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:647)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1456)
        at org.apache.solr.cloud.BasicDistributedZkTest.queryServer(BasicDistributedZkTest.java:1148)
        at org.apache.solr.cloud.BasicDistributedZkTest.testUpdateProcessorsRunOnlyOnce(BasicDistributedZkTest.java:692)
        at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:377)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1126)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1112)
        ... 52 more


FAILED:  org.apache.solr.cloud.DeleteNodeTest.test

Error Message:
Error from server at https://127.0.0.1:45869/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45869/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
        at __randomizedtesting.SeedInfo.seed([BA976DB8DEF25536:32C35262700E38CE]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.DeleteNodeTest.test(DeleteNodeTest.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
Error from server at http://127.0.0.1:41450/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41450/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
        at __randomizedtesting.SeedInfo.seed([BA976DB8DEF25536:17F7D9B3C3CDFD43]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:49)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:45653","node_name":"127.0.0.1:45653_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"c8n_1x3_lf_shard1_replica_n1",           "base_url":"http://127.0.0.1:36970",           "node_name":"127.0.0.1:36970_",           "state":"down",           "type":"NRT"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:39927",           "core":"c8n_1x3_lf_shard1_replica_n3",           "node_name":"127.0.0.1:39927_",           "type":"NRT"},         "core_node3":{           "core":"c8n_1x3_lf_shard1_replica_n2",           "base_url":"http://127.0.0.1:45653",           "node_name":"127.0.0.1:45653_",           "state":"active",           "type":"NRT",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:45653","node_name":"127.0.0.1:45653_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"c8n_1x3_lf_shard1_replica_n1",
          "base_url":"http://127.0.0.1:36970",
          "node_name":"127.0.0.1:36970_",
          "state":"down",
          "type":"NRT"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:39927",
          "core":"c8n_1x3_lf_shard1_replica_n3",
          "node_name":"127.0.0.1:39927_",
          "type":"NRT"},
        "core_node3":{
          "core":"c8n_1x3_lf_shard1_replica_n2",
          "base_url":"http://127.0.0.1:45653",
          "node_name":"127.0.0.1:45653_",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"3",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([BA976DB8DEF25536:32C35262700E38CE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Could not load collection from ZK: unloadcollection

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: unloadcollection
        at __randomizedtesting.SeedInfo.seed([BA976DB8DEF25536:32C35262700E38CE]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1114)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:647)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:277)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/unloadcollection/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1126)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1112)
        ... 48 more




Build Log:
[...truncated 11830 lines...]
   [junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/init-core-data-001
   [junit4]   2> 2575480 WARN  (SUITE-UnloadDistributedZkTest-seed#[BA976DB8DEF25536]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=46 numCloses=46
   [junit4]   2> 2575480 INFO  (SUITE-UnloadDistributedZkTest-seed#[BA976DB8DEF25536]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2575481 INFO  (SUITE-UnloadDistributedZkTest-seed#[BA976DB8DEF25536]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2575481 INFO  (SUITE-UnloadDistributedZkTest-seed#[BA976DB8DEF25536]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2575500 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2575516 INFO  (Thread-2128) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2575516 INFO  (Thread-2128) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2575560 ERROR (Thread-2128) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2575625 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ZkTestServer start zk server on port:43272
   [junit4]   2> 2575928 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2575929 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2575930 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2575943 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2575943 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2575944 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2575944 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2575945 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2575945 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2575946 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2575979 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2575999 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2576588 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2576692 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@53513a7f{/,null,AVAILABLE}
   [junit4]   2> 2576694 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5cafdb1f{HTTP/1.1,[http/1.1]}{127.0.0.1:33278}
   [junit4]   2> 2576694 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server Started @2599493ms
   [junit4]   2> 2576694 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=33278, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/control-001/cores}
   [junit4]   2> 2576711 ERROR (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2576820 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2576820 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2576820 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2576820 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T01:22:20.595Z
   [junit4]   2> 2576878 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2576878 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/control-001/solr.xml
   [junit4]   2> 2576914 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2576931 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43272/solr
   [junit4]   2> 2577601 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2577618 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33278_
   [junit4]   2> 2577618 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.c.Overseer Overseer (id=98660420112285700-127.0.0.1:33278_-n_0000000000) starting
   [junit4]   2> 2577662 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33278_
   [junit4]   2> 2577735 INFO  (OverseerStateUpdate-98660420112285700-127.0.0.1:33278_-n_0000000000) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2578940 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2578977 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2578978 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2578995 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33278_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/control-001/cores
   [junit4]   2> 2579260 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2579261 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43272/solr ready
   [junit4]   2> 2579314 INFO  (qtp1786120536-8314) [n:127.0.0.1:33278_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33278_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2579378 INFO  (OverseerThreadFactory-3533-thread-1-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2579748 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2579749 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2579902 INFO  (zkCallback-1301-thread-1-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2581121 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2581252 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2581858 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2581938 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2581938 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2581938 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2581938 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2581953 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2582011 WARN  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2582155 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2582155 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2582172 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2582172 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2582173 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2582173 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@b41947c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2582174 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2582174 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2582175 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2582192 INFO  (searcherExecutor-3536-thread-1-processing-n:127.0.0.1:33278_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@b41947c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2582192 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578566728775892992
   [junit4]   2> 2582208 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2582209 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2582209 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33278/control_collection_shard1_replica_n1/
   [junit4]   2> 2582209 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2582209 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33278/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2582209 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2582210 INFO  (zkCallback-1301-thread-1-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2582212 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33278/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2582335 INFO  (zkCallback-1301-thread-1-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2582391 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2582392 INFO  (qtp1786120536-8312) [n:127.0.0.1:33278_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2643
   [junit4]   2> 2582460 INFO  (qtp1786120536-8314) [n:127.0.0.1:33278_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2582501 INFO  (zkCallback-1301-thread-1-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2583437 INFO  (OverseerCollectionConfigSetProcessor-98660420112285700-127.0.0.1:33278_-n_0000000000) [n:127.0.0.1:33278_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2583506 INFO  (qtp1786120536-8314) [n:127.0.0.1:33278_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33278_&wt=javabin&version=2} status=0 QTime=4192
   [junit4]   2> 2583560 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2583560 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43272/solr ready
   [junit4]   2> 2583560 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2583571 INFO  (qtp1786120536-8315) [n:127.0.0.1:33278_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2583668 INFO  (OverseerThreadFactory-3533-thread-2-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2583669 WARN  (OverseerThreadFactory-3533-thread-2-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2583999 INFO  (qtp1786120536-8315) [n:127.0.0.1:33278_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2583999 INFO  (qtp1786120536-8315) [n:127.0.0.1:33278_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=427
   [junit4]   2> 2584599 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-1-001 of type NRT
   [junit4]   2> 2584599 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2584639 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c6bb5a3{/,null,AVAILABLE}
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@654da36c{HTTP/1.1,[http/1.1]}{127.0.0.1:33818}
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server Started @2607438ms
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=33818, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-1-001/cores}
   [junit4]   2> 2584640 ERROR (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2584640 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T01:22:28.415Z
   [junit4]   2> 2584709 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2584709 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-1-001/solr.xml
   [junit4]   2> 2584725 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2584761 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43272/solr
   [junit4]   2> 2584926 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2584927 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2584940 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33818_
   [junit4]   2> 2585126 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2585192 INFO  (zkCallback-1308-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2585191 INFO  (zkCallback-1301-thread-2-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2585663 INFO  (OverseerCollectionConfigSetProcessor-98660420112285700-127.0.0.1:33278_-n_0000000000) [n:127.0.0.1:33278_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2586095 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2586196 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2586196 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2586197 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:33818_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-1-001/cores
   [junit4]   2> 2586334 INFO  (qtp1786120536-8318) [n:127.0.0.1:33278_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33818_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2586387 INFO  (OverseerThreadFactory-3533-thread-3-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:33818_ for creating new replica
   [junit4]   2> 2586393 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2586393 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2586593 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2586864 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 2587267 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2587413 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 2587413 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2587414 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2587414 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 2587428 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2587466 WARN  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2587814 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2587814 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2587815 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2587815 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2587816 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2587817 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@ee208b7[collection1_shard2_replica_n0] main]
   [junit4]   2> 2587867 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2587867 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2587868 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2587868 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578566734727610368
   [junit4]   2> 2587876 INFO  (searcherExecutor-3547-thread-1-processing-n:127.0.0.1:33818_ x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@ee208b7[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2587904 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2587904 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2587904 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33818/collection1_shard2_replica_n0/
   [junit4]   2> 2587904 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2587904 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:33818/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 2587904 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2587906 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33818/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 2588042 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2588123 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2588125 INFO  (qtp1241619177-8355) [n:127.0.0.1:33818_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1731
   [junit4]   2> 2588126 INFO  (qtp1786120536-8318) [n:127.0.0.1:33278_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33818_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1792
   [junit4]   2> 2588234 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2588380 INFO  (OverseerCollectionConfigSetProcessor-98660420112285700-127.0.0.1:33278_-n_0000000000) [n:127.0.0.1:33278_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2589136 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-2-001 of type NRT
   [junit4]   2> 2589136 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2589178 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@784606af{/,null,AVAILABLE}
   [junit4]   2> 2589178 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3a13b58f{HTTP/1.1,[http/1.1]}{127.0.0.1:44952}
   [junit4]   2> 2589178 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server Started @2611977ms
   [junit4]   2> 2589178 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=44952, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-2-001/cores}
   [junit4]   2> 2589178 ERROR (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2589198 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2589198 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2589198 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2589198 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T01:22:32.973Z
   [junit4]   2> 2589242 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2589242 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-2-001/solr.xml
   [junit4]   2> 2589334 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2589369 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43272/solr
   [junit4]   2> 2589529 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2589530 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2589544 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44952_
   [junit4]   2> 2589662 INFO  (zkCallback-1319-thread-1-processing-n:127.0.0.1:44952_) [n:127.0.0.1:44952_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2589690 INFO  (zkCallback-1301-thread-1-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2589690 INFO  (zkCallback-1308-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2589698 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2590223 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2590293 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2590294 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2590295 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [n:127.0.0.1:44952_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-2-001/cores
   [junit4]   2> 2590569 INFO  (qtp467506155-8385) [n:127.0.0.1:44952_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:44952_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2590572 INFO  (OverseerThreadFactory-3533-thread-4-processing-n:127.0.0.1:33278_) [n:127.0.0.1:33278_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:44952_ for creating new replica
   [junit4]   2> 2590627 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2590627 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2590646 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2590866 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2590976 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema [collection1_shard1_replica_n0] Schema name=test
   [junit4]   2> 2592050 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2592105 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 2592105 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n0' (registry 'solr.core.collection1.shard1.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1
   [junit4]   2> 2592106 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2592106 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [[collection1_shard1_replica_n0] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-2-001/cores/collection1_shard1_replica_n0], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-2-001/cores/collection1_shard1_replica_n0/data/]
   [junit4]   2> 2592124 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2592140 WARN  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2592431 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2592431 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2592432 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2592432 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2592433 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2592433 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@48f7cd8c[collection1_shard1_replica_n0] main]
   [junit4]   2> 2592434 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2592434 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2592435 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2592469 INFO  (searcherExecutor-3558-thread-1-processing-n:127.0.0.1:44952_ x:collection1_shard1_replica_n0 s:shard1 c:collection1) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [collection1_shard1_replica_n0] Registered new searcher Searcher@48f7cd8c[collection1_shard1_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2592469 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578566739552108544
   [junit4]   2> 2592506 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2592506 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2592506 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44952/collection1_shard1_replica_n0/
   [junit4]   2> 2592506 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2592507 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy http://127.0.0.1:44952/collection1_shard1_replica_n0/ has no replicas
   [junit4]   2> 2592507 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2592525 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44952/collection1_shard1_replica_n0/ shard1
   [junit4]   2> 2592635 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2592647 INFO  (zkCallback-1319-thread-1-processing-n:127.0.0.1:44952_) [n:127.0.0.1:44952_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2592687 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2592689 INFO  (qtp467506155-8387) [n:127.0.0.1:44952_ c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2062
   [junit4]   2> 2592724 INFO  (qtp467506155-8385) [n:127.0.0.1:44952_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:44952_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=2155
   [junit4]   2> 2592797 INFO  (zkCallback-1313-thread-1-processing-n:127.0.0.1:33818_) [n:127.0.0.1:33818_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2592813 INFO  (zkCallback-1319-thread-1-processing-n:127.0.0.1:44952_) [n:127.0.0.1:44952_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2593326 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-3-001 of type NRT
   [junit4]   2> 2593341 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2593363 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3f087db9{/,null,AVAILABLE}
   [junit4]   2> 2593363 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@368aecda{HTTP/1.1,[http/1.1]}{127.0.0.1:43971}
   [junit4]   2> 2593363 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.e.j.s.Server Started @2616162ms
   [junit4]   2> 2593363 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=43971, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-3-001/cores}
   [junit4]   2> 2593364 ERROR (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2593423 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2593423 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2593423 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2593423 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T01:22:37.198Z
   [junit4]   2> 2593498 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2593498 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_BA976DB8DEF25536-001/shard-3-001/solr.xml
   [junit4]   2> 2593514 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5e88fef1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2593606 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA976DB8DEF25536]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43272/solr
   [junit4]   2> 2593710 INFO  (TEST-UnloadDistributedZkTest.test-seed#[BA

[...truncated too long message...]

e/test-lib/antlr-2.7.7.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/api-all-1.0.0-M20.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/bcprov-jdk15-1.45.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/byte-buddy-1.6.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/commons-math3-3.4.1.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/ehcache-core-2.4.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/jersey-core-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/jersey-server-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/mina-core-2.0.0-M5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/mockito-core-2.6.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/netty-all-4.0.36.Final.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/test-lib/objenesis-2.5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/contrib/solr-analysis-extras/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/lucene/build/queryparser/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/contrib/analysis-extras/lib/icu4j-59.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/temp/junit4-J1-20170915_003900_4715623703793282963808.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/temp/junit4-J1-20170915_003900_4717735580437238338420.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Forked process returned with error code: 137.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/build.xml:810: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/build.xml:754: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/build.xml:59: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build.xml:267: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/common-build.xml:549: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/lucene/common-build.xml:1455: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/lucene/common-build.xml:985: At least one slave process threw an exception, first: Forked process returned with error code: 137.

Total time: 280 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-7.0 #133 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.0 - Build # 135 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.0/135/

3 tests failed.
FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([72D8756A01CA57A4:FA8C4AB0AF363A5C]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:111)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudPivotFacet.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([72D8756A01CA57A4:FA8C4AB0AF363A5C]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1114)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:647)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.getTotalReplicas(AbstractFullDistribZkTestBase.java:488)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:441)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:334)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1126)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1112)
        ... 38 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:39101","node_name":"127.0.0.1:39101_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/18)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"c8n_1x3_lf_shard1_replica_n1",           "base_url":"http://127.0.0.1:43855",           "node_name":"127.0.0.1:43855_",           "state":"down",           "type":"NRT"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:34546",           "core":"c8n_1x3_lf_shard1_replica_n3",           "node_name":"127.0.0.1:34546_",           "type":"NRT"},         "core_node3":{           "core":"c8n_1x3_lf_shard1_replica_n2",           "base_url":"http://127.0.0.1:39101",           "node_name":"127.0.0.1:39101_",           "state":"active",           "type":"NRT",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:39101","node_name":"127.0.0.1:39101_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"c8n_1x3_lf_shard1_replica_n1",
          "base_url":"http://127.0.0.1:43855",
          "node_name":"127.0.0.1:43855_",
          "state":"down",
          "type":"NRT"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:34546",
          "core":"c8n_1x3_lf_shard1_replica_n3",
          "node_name":"127.0.0.1:34546_",
          "type":"NRT"},
        "core_node3":{
          "core":"c8n_1x3_lf_shard1_replica_n2",
          "base_url":"http://127.0.0.1:39101",
          "node_name":"127.0.0.1:39101_",
          "state":"active",
          "type":"NRT",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"3",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([72D8756A01CA57A4:FA8C4AB0AF363A5C]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11082 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/init-core-data-001
   [junit4]   2> 363112 WARN  (SUITE-MultiThreadedOCPTest-seed#[72D8756A01CA57A4]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=32 numCloses=32
   [junit4]   2> 363112 INFO  (SUITE-MultiThreadedOCPTest-seed#[72D8756A01CA57A4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 363113 INFO  (SUITE-MultiThreadedOCPTest-seed#[72D8756A01CA57A4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 363148 INFO  (SUITE-MultiThreadedOCPTest-seed#[72D8756A01CA57A4]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /j_lfl/kx
   [junit4]   2> 363150 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 363168 INFO  (Thread-363) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 363168 INFO  (Thread-363) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 363184 ERROR (Thread-363) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 363277 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkTestServer start zk server on port:38315
   [junit4]   2> 363367 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 363368 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 363369 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 363387 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 363388 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 363389 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 363390 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 363390 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 363391 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 363405 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 363406 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 363407 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e8713cf400001, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 363442 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e8713cf400002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 363443 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 363865 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 364025 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f546921{/j_lfl/kx,null,AVAILABLE}
   [junit4]   2> 364026 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4f259fe5{SSL,[ssl, http/1.1]}{127.0.0.1:42076}
   [junit4]   2> 364026 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server Started @367077ms
   [junit4]   2> 364026 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/tempDir-001/control/data, hostContext=/j_lfl/kx, hostPort=42076, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/control-001/cores}
   [junit4]   2> 364027 ERROR (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 364027 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 364027 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 364027 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 364027 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T19:46:16.857Z
   [junit4]   2> 364059 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 364059 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/control-001/solr.xml
   [junit4]   2> 364097 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 364168 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38315/solr
   [junit4]   2> 365078 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 365111 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42076_j_lfl%2Fkx
   [junit4]   2> 365112 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.Overseer Overseer (id=98664760999936005-127.0.0.1:42076_j_lfl%2Fkx-n_0000000000) starting
   [junit4]   2> 365188 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42076_j_lfl%2Fkx
   [junit4]   2> 365208 INFO  (zkCallback-221-thread-1-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 365481 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 365573 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 365592 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 365594 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/control-001/cores
   [junit4]   2> 365965 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 365965 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38315/solr ready
   [junit4]   2> 366428 INFO  (qtp799432183-1432) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42076_j_lfl%252Fkx&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 366525 INFO  (OverseerThreadFactory-787-thread-1-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 366704 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 366704 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 366875 INFO  (zkCallback-221-thread-1-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 367966 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 368040 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 369046 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 369161 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 369174 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 369175 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 369175 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 369177 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=50.9775390625, floorSegmentMB=1.11328125, forceMergeDeletesPctAllowed=22.771542192850557, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 369265 WARN  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 369726 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 369726 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 369728 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 369728 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 369729 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.809913919464824]
   [junit4]   2> 369729 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@68294620[control_collection_shard1_replica_n1] main]
   [junit4]   2> 369742 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 369763 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 369763 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 369764 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578636182913286144
   [junit4]   2> 369766 INFO  (searcherExecutor-790-thread-1-processing-n:127.0.0.1:42076_j_lfl%2Fkx x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@68294620[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 369782 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 369782 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 369782 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42076/j_lfl/kx/control_collection_shard1_replica_n1/
   [junit4]   2> 369782 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 369782 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:42076/j_lfl/kx/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 369782 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 369821 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42076/j_lfl/kx/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 369852 INFO  (zkCallback-221-thread-2-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 369966 INFO  (zkCallback-221-thread-2-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 369982 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 369983 INFO  (qtp799432183-1428) [n:127.0.0.1:42076_j_lfl%2Fkx c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3279
   [junit4]   2> 370066 INFO  (qtp799432183-1432) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 370112 INFO  (zkCallback-221-thread-2-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 370626 INFO  (OverseerCollectionConfigSetProcessor-98664760999936005-127.0.0.1:42076_j_lfl%2Fkx-n_0000000000) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 371068 INFO  (qtp799432183-1432) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42076_j_lfl%252Fkx&wt=javabin&version=2} status=0 QTime=4639
   [junit4]   2> 371227 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 371228 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38315/solr ready
   [junit4]   2> 371228 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 371244 INFO  (qtp799432183-1433) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 371283 INFO  (OverseerThreadFactory-787-thread-2-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 371284 WARN  (OverseerThreadFactory-787-thread-2-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 371739 INFO  (qtp799432183-1433) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 371740 INFO  (qtp799432183-1433) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=496
   [junit4]   2> 372786 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-1-001 of type NRT
   [junit4]   2> 372806 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 372856 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6f9dc68{/j_lfl/kx,null,AVAILABLE}
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6352d052{SSL,[ssl, http/1.1]}{127.0.0.1:46628}
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server Started @375907ms
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/j_lfl/kx, hostPort=46628, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-1-001/cores}
   [junit4]   2> 372857 ERROR (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 372857 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T19:46:25.687Z
   [junit4]   2> 372892 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 372892 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-1-001/solr.xml
   [junit4]   2> 372966 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 373009 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38315/solr
   [junit4]   2> 373175 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 373229 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 373231 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46628_j_lfl%2Fkx
   [junit4]   2> 373376 INFO  (zkCallback-228-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 373376 INFO  (zkCallback-221-thread-3-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 373376 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 373464 INFO  (OverseerCollectionConfigSetProcessor-98664760999936005-127.0.0.1:42076_j_lfl%2Fkx-n_0000000000) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 373829 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 373903 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 373903 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 373905 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-1-001/cores
   [junit4]   2> 374363 INFO  (qtp295939862-1475) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:46628_j_lfl%252Fkx&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 374542 INFO  (OverseerThreadFactory-787-thread-3-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:46628_j_lfl%2Fkx for creating new replica
   [junit4]   2> 374616 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 374616 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 374727 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 375765 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 375926 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 376824 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 376919 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 376919 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 376919 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 376920 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-1-001/cores/collection1_shard2_replica_n0/data/]
   [junit4]   2> 376954 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=50.9775390625, floorSegmentMB=1.11328125, forceMergeDeletesPctAllowed=22.771542192850557, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 376956 WARN  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 377539 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 377540 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 377541 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 377541 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 377542 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.809913919464824]
   [junit4]   2> 377542 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@75c4122[collection1_shard2_replica_n0] main]
   [junit4]   2> 377556 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 377556 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 377557 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 377557 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578636191084838912
   [junit4]   2> 377574 INFO  (searcherExecutor-801-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@75c4122[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 377615 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 377615 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 377615 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:46628/j_lfl/kx/collection1_shard2_replica_n0/
   [junit4]   2> 377615 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 377615 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:46628/j_lfl/kx/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 377615 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 377630 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:46628/j_lfl/kx/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 377739 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 378088 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 378124 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 378125 INFO  (qtp295939862-1477) [n:127.0.0.1:46628_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3509
   [junit4]   2> 378142 INFO  (qtp295939862-1475) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:46628_j_lfl%252Fkx&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=3779
   [junit4]   2> 378580 INFO  (OverseerCollectionConfigSetProcessor-98664760999936005-127.0.0.1:42076_j_lfl%2Fkx-n_0000000000) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 378741 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 379203 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-2-001 of type NRT
   [junit4]   2> 379204 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 379217 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3794b30d{/j_lfl/kx,null,AVAILABLE}
   [junit4]   2> 379218 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6ecf7596{SSL,[ssl, http/1.1]}{127.0.0.1:34322}
   [junit4]   2> 379218 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server Started @382269ms
   [junit4]   2> 379218 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/j_lfl/kx, hostPort=34322, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-2-001/cores}
   [junit4]   2> 379218 ERROR (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 379237 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 379237 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 379290 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 379290 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T19:46:32.120Z
   [junit4]   2> 379396 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 379397 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-2-001/solr.xml
   [junit4]   2> 379450 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 379522 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38315/solr
   [junit4]   2> 379986 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 379987 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 380001 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34322_j_lfl%2Fkx
   [junit4]   2> 380003 INFO  (zkCallback-221-thread-3-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 380070 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:34322_j_lfl%2Fkx) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 380095 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 380107 INFO  (zkCallback-228-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 380830 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 380899 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 380899 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 380913 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-2-001/cores
   [junit4]   2> 381399 INFO  (qtp1739768730-1499) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34322_j_lfl%252Fkx&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 381434 INFO  (OverseerThreadFactory-787-thread-4-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:34322_j_lfl%2Fkx for creating new replica
   [junit4]   2> 381559 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 381559 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 381564 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 381645 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 381773 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema [collection1_shard1_replica_n0] Schema name=test
   [junit4]   2> 382518 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 382576 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 382576 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n0' (registry 'solr.core.collection1.shard1.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 382577 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 382577 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [[collection1_shard1_replica_n0] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-2-001/cores/collection1_shard1_replica_n0], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-2-001/cores/collection1_shard1_replica_n0/data/]
   [junit4]   2> 382591 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=50.9775390625, floorSegmentMB=1.11328125, forceMergeDeletesPctAllowed=22.771542192850557, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 382613 WARN  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 382851 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 382852 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 382853 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 382853 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 382854 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.809913919464824]
   [junit4]   2> 382854 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@1262c838[collection1_shard1_replica_n0] main]
   [junit4]   2> 382868 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 382868 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 382869 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 382869 INFO  (searcherExecutor-812-thread-1-processing-n:127.0.0.1:34322_j_lfl%2Fkx x:collection1_shard1_replica_n0 s:shard1 c:collection1) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SolrCore [collection1_shard1_replica_n0] Registered new searcher Searcher@1262c838[collection1_shard1_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 382870 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578636196655923200
   [junit4]   2> 382961 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 382961 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 382961 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:34322/j_lfl/kx/collection1_shard1_replica_n0/
   [junit4]   2> 382961 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 382961 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:34322/j_lfl/kx/collection1_shard1_replica_n0/ has no replicas
   [junit4]   2> 382961 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 382963 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:34322/j_lfl/kx/collection1_shard1_replica_n0/ shard1
   [junit4]   2> 383094 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 383098 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:34322_j_lfl%2Fkx) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 383123 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 383132 INFO  (qtp1739768730-1505) [n:127.0.0.1:34322_j_lfl%2Fkx c:collection1 s:shard1  x:collection1_shard1_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n0&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1573
   [junit4]   2> 383197 INFO  (qtp1739768730-1499) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34322_j_lfl%252Fkx&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1797
   [junit4]   2> 383241 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:34322_j_lfl%2Fkx) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 383241 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 383448 INFO  (OverseerCollectionConfigSetProcessor-98664760999936005-127.0.0.1:42076_j_lfl%2Fkx-n_0000000000) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 384276 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-3-001 of type NRT
   [junit4]   2> 384330 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 384431 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@301951c1{/j_lfl/kx,null,AVAILABLE}
   [junit4]   2> 384447 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@129d9f54{SSL,[ssl, http/1.1]}{127.0.0.1:46486}
   [junit4]   2> 384447 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.Server Started @387498ms
   [junit4]   2> 384447 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/j_lfl/kx, hostPort=46486, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-3-001/cores}
   [junit4]   2> 384447 ERROR (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 384482 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 384482 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 384482 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 384482 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-15T19:46:37.312Z
   [junit4]   2> 384594 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 384594 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-3-001/solr.xml
   [junit4]   2> 384612 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 384637 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38315/solr
   [junit4]   2> 384809 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 384822 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 384861 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46486_j_lfl%2Fkx
   [junit4]   2> 384872 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 384872 INFO  (zkCallback-221-thread-2-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 384872 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:34322_j_lfl%2Fkx) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 384872 INFO  (zkCallback-228-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 384980 INFO  (zkCallback-245-thread-1-processing-n:127.0.0.1:46486_j_lfl%2Fkx) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 385798 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 385852 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 385852 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 385853 INFO  (TEST-MultiThreadedOCPTest.test-seed#[72D8756A01CA57A4]) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-3-001/cores
   [junit4]   2> 386456 INFO  (qtp1739768730-1502) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:46486_j_lfl%252Fkx&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 386459 INFO  (OverseerThreadFactory-787-thread-5-processing-n:127.0.0.1:42076_j_lfl%2Fkx) [n:127.0.0.1:42076_j_lfl%2Fkx    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:46486_j_lfl%2Fkx for creating new replica
   [junit4]   2> 386763 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 386764 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 386781 INFO  (zkCallback-233-thread-1-processing-n:127.0.0.1:46628_j_lfl%2Fkx) [n:127.0.0.1:46628_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 386801 INFO  (zkCallback-239-thread-1-processing-n:127.0.0.1:34322_j_lfl%2Fkx) [n:127.0.0.1:34322_j_lfl%2Fkx    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 386893 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 387095 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 388171 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 388341 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 388341 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@54878279
   [junit4]   2> 388341 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 388341 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-3-001/cores/collection1_shard2_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_72D8756A01CA57A4-001/shard-3-001/cores/collection1_shard2_replica_n1/data/]
   [junit4]   2> 388377 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=50.9775390625, floorSegmentMB=1.11328125, forceMergeDeletesPctAllowed=22.771542192850557, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 388412 WARN  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 388747 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 388747 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 388748 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 388748 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 388765 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.809913919464824]
   [junit4]   2> 388766 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@d1688e9[collection1_shard2_replica_n1] main]
   [junit4]   2> 388830 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 388850 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 388851 INFO  (qtp1648183929-1534) [n:127.0.0.1:46486_j_lfl%2Fkx c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 388851 INFO  (qtp1648183929-1534) [n:127.0.0.1:4

[...truncated too long message...]

 null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@21cfc2a8
   [junit4]   2> 1425127 INFO  (TEST-TestCloudPivotFacet.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1425161 INFO  (TEST-TestCloudPivotFacet.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.Overseer Overseer (id=98664827723186194-127.0.0.1:40753_-n_0000000004) closing
   [junit4]   2> 1425161 INFO  (OverseerStateUpdate-98664827723186194-127.0.0.1:40753_-n_0000000004) [n:127.0.0.1:40753_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40753_
   [junit4]   2> 1425575 WARN  (zkCallback-1042-thread-1-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_ c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Closed, skipping sync up.
   [junit4]   2> 1425575 INFO  (zkCallback-1042-thread-1-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_ c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1425575 INFO  (zkCallback-1042-thread-1-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_ c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@776415a5
   [junit4]   2> 1425575 INFO  (zkCallback-1042-thread-1-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_ c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_n1, tag=2003047845
   [junit4]   2> 1425575 INFO  (zkCallback-1042-thread-1-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_ c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@228fe78a: rootName = null, domain = solr.core.collection1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@eb07bdb
   [junit4]   2> 1425648 INFO  (zkCallback-1042-thread-1-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_ c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=2003047845
   [junit4]   2> 1426662 WARN  (zkCallback-1042-thread-2-processing-n:127.0.0.1:40753_) [n:127.0.0.1:40753_    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1426663 INFO  (TEST-TestCloudPivotFacet.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@15d50865{HTTP/1.1,[http/1.1]}{127.0.0.1:?}
   [junit4]   2> 1426663 INFO  (TEST-TestCloudPivotFacet.test-seed#[72D8756A01CA57A4]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1aa57b4c{/,null,UNAVAILABLE}
   [junit4]   2> 1426664 ERROR (TEST-TestCloudPivotFacet.test-seed#[72D8756A01CA57A4]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1426664 INFO  (TEST-TestCloudPivotFacet.test-seed#[72D8756A01CA57A4]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41738 41738
   [junit4]   2> 1426925 INFO  (Thread-1850) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41738 41738
   [junit4]   2> 1426925 WARN  (Thread-1850) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/aliases.json
   [junit4]   2> 5 /solr/security.json
   [junit4]   2> 5 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/clusterstate.json
   [junit4]   2> 7 /solr/clusterprops.json
   [junit4]   2> 4 /solr/collections/collection1/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/live_nodes
   [junit4]   2> 7 /solr/collections
   [junit4]   2> 4 /solr/overseer/collection-queue-work
   [junit4]   2> 3 /solr/overseer/queue
   [junit4]   2>
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCloudPivotFacet -Dtests.method=test -Dtests.seed=72D8756A01CA57A4 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=hi-IN -Dtests.timezone=Poland -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   45.9s J1 | TestCloudPivotFacet.test <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
   [junit4]    > at __randomizedtesting.SeedInfo.seed([72D8756A01CA57A4:FA8C4AB0AF363A5C]:0)
   [junit4]    > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1114)
   [junit4]    > at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:647)
   [junit4]    > at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.getTotalReplicas(AbstractFullDistribZkTestBase.java:488)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:441)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:334)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
   [junit4]    > at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]    > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]    > at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]    > at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]    > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    > at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]    > at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1126)
   [junit4]    > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1112)
   [junit4]    > ... 38 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.0/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudPivotFacet_72D8756A01CA57A4-001
   [junit4]   2> Sep 15, 2017 8:03:59 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=20, maxMBSortInHeap=5.8954718431129525, sim=RandomSimilarity(queryNorm=false): {}, locale=hi-IN, timezone=Poland
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=212304504,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [RestartWhileUpdatingTest, TestSystemCollAutoCreate, TolerantUpdateProcessorTest, LeaderElectionContextKeyTest, TestExclusionRuleCollectionAccess, SecurityConfHandlerTest, SpatialFilterTest, SolrMetricsIntegrationTest, TestSQLHandlerNonCloud, AutoScalingHandlerTest, SuggesterTSTTest, HdfsDirectoryFactoryTest, TestCoreAdminApis, QueryParsingTest, HdfsRecoveryZkTest, TestClusterProperties, TestUpdate, TestDynamicFieldResource, TestHalfAndHalfDocValues, DistribJoinFromCollectionTest, TestFreeTextSuggestions, TestCloudDeleteByQuery, AsyncCallRequestStatusResponseTest, BitVectorTest, TestFieldCacheReopen, DataDrivenBlockJoinTest, TestGeoJSONResponseWriter, TestSha256AuthenticationProvider, TestSegmentSorting, TestFieldCacheVsDocValues, TestDynamicLoading, TestRequestForwarding, CloudMLTQParserTest, FileBasedSpellCheckerTest, TestDocBasedVersionConstraints, BlockDirectoryTest, TestQueryTypes, SoftAutoCommitTest, SortSpecParsingTest, DistributedQueryComponentCustomSortTest, TestNumericRangeQuery32, CoreAdminHandlerTest, TestSolrQueryResponse, TestDFISimilarityFactory, InfoHandlerTest, CollectionsAPIDistributedZkTest, TestScoreJoinQPNoScore, DistributedFacetExistsSmallTest, ClusterStateUpdateTest, TestAtomicUpdateErrorCases, CdcrVersionReplicationTest, TestHttpShardHandlerFactory, BadComponentTest, OverseerModifyCollectionTest, TestPerFieldSimilarity, TestManagedSchemaThreadSafety, TestFastLRUCache, TestValueSourceCache, BasicFunctionalityTest, TestMaxScoreQueryParser, ChaosMonkeySafeLeaderTest, TestSimpleTrackingShardHandler, TestPostingsSolrHighlighter, LargeFieldTest, RuleEngineTest, TestCharFilters, SolrJmxReporterCloudTest, AnalysisAfterCoreReloadTest, DOMUtilTest, NoCacheHeaderTest, AlternateDirectoryTest, ResourceLoaderTest, HdfsBasicDistributedZkTest, UpdateRequestProcessorFactoryTest, ResponseHeaderTest, CdcrReplicationDistributedZkTest, TestCloudPivotFacet]
   [junit4] Completed [185/729 (3!)] on J1 in 46.30s, 1 test, 1 error <<< FAILURES!

[...truncated 45218 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]