[JENKINS] Lucene-Solr-Tests-7.x - Build # 271 - Still Failing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 271 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/271/

All tests passed

Build Log:
[...truncated 11834 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J1-20171205_201306_6466594861230560110653.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] #
   [junit4] # There is insufficient memory for the Java Runtime Environment to continue.
   [junit4] # Native memory allocation (mmap) failed to map 524288 bytes for committing reserved memory.
   [junit4] # An error report file with more information is saved as:
   [junit4] # /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/hs_err_pid31688.log
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J1-20171205_201306_64686331757623105717.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000fff80000, 524288, 0) failed; error='Cannot allocate memory' (errno=12)
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452958554149442269497.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] #
   [junit4] # There is insufficient memory for the Java Runtime Environment to continue.
   [junit4] # Native memory allocation (mmap) failed to map 524288 bytes for committing reserved memory.
   [junit4] # An error report file with more information is saved as:
   [junit4] # /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/hs_err_pid31689.log
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452501446359170765425.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000fff00000, 524288, 0) failed; error='Cannot allocate memory' (errno=12)
   [junit4] <<< JVM J0: EOF ----

[...truncated 1646 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_144/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=933C5EC5DC8FD0FB -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.2.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=7.2.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x -Djava.security.egd=file:/dev/./urandom -Djunit4.childvm.cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/test-framework/lib/junit4-ant-2.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/common/lucene-analyzers-common-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/codecs/lucene-codecs-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/backward-codecs/lucene-backward-codecs-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/highlighter/lucene-highlighter-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/memory/lucene-memory-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/misc/lucene-misc-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/spatial-extras/lucene-spatial-extras-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/spatial3d/lucene-spatial3d-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/expressions/lucene-expressions-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/suggest/lucene-suggest-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/grouping/lucene-grouping-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/queries/lucene-queries-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/queryparser/lucene-queryparser-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/join/lucene-join-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/sandbox/lucene-sandbox-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/classification/lucene-classification-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/asm-5.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/asm-commons-5.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/avatica-core-1.10.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/caffeine-2.4.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/calcite-core-1.13.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/calcite-linq4j-1.13.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-cli-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-codec-1.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-compiler-2.7.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-configuration-1.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-exec-1.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-fileupload-1.3.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-io-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-lang-2.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-lang3-3.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/curator-client-2.8.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/curator-framework-2.8.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/curator-recipes-2.8.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/dom4j-1.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/eigenbase-properties-1.1.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/guava-14.0.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-annotations-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-auth-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-common-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-hdfs-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hppc-0.7.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-core-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/janino-2.7.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/joda-time-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/log4j-1.2.17.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/org.restlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/protobuf-java-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/slf4j-log4j12-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/spatial4j-0.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/t-digest-3.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/commons-io-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/httpclient-4.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/httpcore-4.4.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/httpmime-4.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/noggit-0.8.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/slf4j-api-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/stax2-api-3.1.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/zookeeper-3.4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/gmetric4j-1.0.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/javax.servlet-api-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-continuation-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-deploy-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-http-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-io-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-jmx-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-rewrite-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-security-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-server-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-servlet-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-servlets-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-util-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-webapp-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-xml-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-core-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-ganglia-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-graphite-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-jetty9-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-jvm-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/example/example-DIH/solr/db/lib/hsqldb-2.4.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/antlr-2.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/byte-buddy-1.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/mockito-core-2.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/objenesis-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/icu/lucene-analyzers-icu-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/backward-codecs/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/contrib/analysis-extras/lib/icu4j-59.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.3.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6451458396758027505394.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452993565271859699570.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452958554149442269497.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452501446359170765425.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_144/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=933C5EC5DC8FD0FB -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.2.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=7.2.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x -Djava.security.egd=file:/dev/./urandom -Djunit4.childvm.cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/test-framework/lib/junit4-ant-2.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/common/lucene-analyzers-common-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/codecs/lucene-codecs-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/backward-codecs/lucene-backward-codecs-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/highlighter/lucene-highlighter-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/memory/lucene-memory-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/misc/lucene-misc-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/spatial-extras/lucene-spatial-extras-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/spatial3d/lucene-spatial3d-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/expressions/lucene-expressions-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/suggest/lucene-suggest-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/grouping/lucene-grouping-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/queries/lucene-queries-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/queryparser/lucene-queryparser-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/join/lucene-join-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/sandbox/lucene-sandbox-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/classification/lucene-classification-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/asm-5.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/asm-commons-5.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/avatica-core-1.10.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/caffeine-2.4.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/calcite-core-1.13.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/calcite-linq4j-1.13.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-cli-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-codec-1.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-compiler-2.7.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-configuration-1.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-exec-1.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-fileupload-1.3.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-io-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-lang-2.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/commons-lang3-3.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/curator-client-2.8.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/curator-framework-2.8.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/curator-recipes-2.8.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/dom4j-1.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/eigenbase-properties-1.1.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/guava-14.0.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-annotations-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-auth-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-common-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hadoop-hdfs-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/hppc-0.7.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-core-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/janino-2.7.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/joda-time-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/log4j-1.2.17.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/org.restlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/protobuf-java-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/slf4j-log4j12-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/spatial4j-0.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/lib/t-digest-3.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/commons-io-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/httpclient-4.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/httpcore-4.4.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/httpmime-4.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/noggit-0.8.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/slf4j-api-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/stax2-api-3.1.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/lib/zookeeper-3.4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/gmetric4j-1.0.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/javax.servlet-api-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-continuation-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-deploy-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-http-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-io-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-jmx-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-rewrite-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-security-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-server-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-servlet-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-servlets-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-util-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-webapp-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/jetty-xml-9.3.20.v20170531.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-core-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-ganglia-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-graphite-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-jetty9-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/server/lib/metrics-jvm-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/example/example-DIH/solr/db/lib/hsqldb-2.4.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/antlr-2.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/byte-buddy-1.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/mockito-core-2.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/test-lib/objenesis-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/analysis/icu/lucene-analyzers-icu-7.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/backward-codecs/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/contrib/analysis-extras/lib/icu4j-59.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.3.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J1-20171205_201306_6467520619521443362231.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J1-20171205_201306_6465993096155578710230.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J1-20171205_201306_6466594861230560110653.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J1-20171205_201306_64686331757623105717.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:779: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build.xml:262: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/common-build.xml:556: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:1512: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:1038: At least one slave process threw an exception, first: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452958554149442269497.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/temp/junit4-J0-20171205_201306_6452501446359170765425.syserr

Total time: 342 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-7.x #266 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 272 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/272/

8 tests failed.
FAILED:  org.apache.solr.cloud.TestPullReplica.testAddRemovePullReplica

Error Message:
Could not load collection from ZK: pull_replica_test_add_remove_pull_replica

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: pull_replica_test_add_remove_pull_replica
        at __randomizedtesting.SeedInfo.seed([A299E67BDB527C2D:D1A4516EC3E56FA9]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:365)
        at org.apache.solr.cloud.TestPullReplica.assertNumberOfReplicas(TestPullReplica.java:542)
        at org.apache.solr.cloud.TestPullReplica.testAddRemovePullReplica(TestPullReplica.java:271)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/pull_replica_test_add_remove_pull_replica/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 43 more


FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([A299E67BDB527C2D:F5B01DCE00A09E36]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi(AutoScalingHandlerTest.java:724)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testExecutorStream

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A759FBB4412575E9:85997A4F624F5FF9]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testExecutorStream(StreamExpressionTest.java:8443)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamExpressionTest

Error Message:
11 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamExpressionTest:     1) Thread[id=1659, name=zkConnectionManagerCallback-311-thread-1-EventThread, state=WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    2) Thread[id=1433, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4535, name=zkCallback-310-thread-7, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=1415, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=1421, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=1418, name=zkConnectionManagerCallback-311-thread-1, state=WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=4534, name=zkCallback-310-thread-6, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=1432, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=1658, name=zkConnectionManagerCallback-311-thread-1-SendThread(127.0.0.1:36987), state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)   10) Thread[id=3723, name=zkCallback-310-thread-5, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=1422, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 11 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamExpressionTest:
   1) Thread[id=1659, name=zkConnectionManagerCallback-311-thread-1-EventThread, state=WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   2) Thread[id=1433, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4535, name=zkCallback-310-thread-7, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=1415, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=1421, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=1418, name=zkConnectionManagerCallback-311-thread-1, state=WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=4534, name=zkCallback-310-thread-6, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=1432, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=1658, name=zkConnectionManagerCallback-311-thread-1-SendThread(127.0.0.1:36987), state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
  10) Thread[id=3723, name=zkCallback-310-thread-5, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  11) Thread[id=1422, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A759FBB4412575E9]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamExpressionTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4535, name=zkCallback-310-thread-7, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=1418, name=zkConnectionManagerCallback-311-thread-1, state=WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=4534, name=zkCallback-310-thread-6, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=1658, name=zkConnectionManagerCallback-311-thread-1-SendThread(127.0.0.1:36987), state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    5) Thread[id=3723, name=zkCallback-310-thread-5, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4535, name=zkCallback-310-thread-7, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=1418, name=zkConnectionManagerCallback-311-thread-1, state=WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=4534, name=zkCallback-310-thread-6, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=1658, name=zkConnectionManagerCallback-311-thread-1-SendThread(127.0.0.1:36987), state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   5) Thread[id=3723, name=zkCallback-310-thread-5, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A759FBB4412575E9]:0)


FAILED:  org.apache.solr.client.solrj.request.TestV2Request.testHttpSolrClient

Error Message:
Error from server at http://127.0.0.1:36061/solr: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:41766/solr/test_shard2_replica_n4/, http://127.0.0.1:37763/solr/test_shard2_replica_n6/, http://127.0.0.1:40789/solr/test_shard1_replica_n1/, http://127.0.0.1:36061/solr/test_shard1_replica_n2/]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:36061/solr: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:41766/solr/test_shard2_replica_n4/, http://127.0.0.1:37763/solr/test_shard2_replica_n6/, http://127.0.0.1:40789/solr/test_shard1_replica_n1/, http://127.0.0.1:36061/solr/test_shard1_replica_n2/]
        at __randomizedtesting.SeedInfo.seed([A759FBB4412575E9:7F41E39EE01167CE]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:829)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.client.solrj.request.TestV2Request.assertSuccess(TestV2Request.java:42)
        at org.apache.solr.client.solrj.request.TestV2Request.doTest(TestV2Request.java:86)
        at org.apache.solr.client.solrj.request.TestV2Request.testHttpSolrClient(TestV2Request.java:62)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.request.TestV2Request.testCloudSolrClient

Error Message:
Error from server at http://127.0.0.1:36061/solr: collection already exists: test

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:36061/solr: collection already exists: test
        at __randomizedtesting.SeedInfo.seed([A759FBB4412575E9:3EAF7004E496FD82]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:829)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.client.solrj.request.TestV2Request.assertSuccess(TestV2Request.java:42)
        at org.apache.solr.client.solrj.request.TestV2Request.doTest(TestV2Request.java:72)
        at org.apache.solr.client.solrj.request.TestV2Request.testCloudSolrClient(TestV2Request.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestV2Request

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:     1) Thread[id=2357, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
   1) Thread[id=2357, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([A759FBB4412575E9]:0)




Build Log:
[...truncated 12818 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_A299E67BDB527C2D-001/init-core-data-001
   [junit4]   2> 3591733 WARN  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=13 numCloses=13
   [junit4]   2> 3591949 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3591951 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3591951 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3591952 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_A299E67BDB527C2D-001/tempDir-001
   [junit4]   2> 3591952 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3591977 INFO  (Thread-4222) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3591977 INFO  (Thread-4222) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3592087 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:44942
   [junit4]   2> 3592088 ERROR (Thread-4222) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3592144 INFO  (zkConnectionManagerCallback-2274-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3592180 INFO  (jetty-launcher-2271-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3592182 INFO  (jetty-launcher-2271-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3592182 INFO  (jetty-launcher-2271-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d046dc9{/solr,null,AVAILABLE}
   [junit4]   2> 3592184 INFO  (jetty-launcher-2271-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7d34116e{SSL,[ssl, http/1.1]}{127.0.0.1:42415}
   [junit4]   2> 3592184 INFO  (jetty-launcher-2271-thread-2) [    ] o.e.j.s.Server Started @3608845ms
   [junit4]   2> 3592184 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42415}
   [junit4]   2> 3592184 ERROR (jetty-launcher-2271-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3592197 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.0
   [junit4]   2> 3592197 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3592197 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3592197 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-06T10:25:16.276Z
   [junit4]   2> 3592198 INFO  (jetty-launcher-2271-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@54ea76a8{/solr,null,AVAILABLE}
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2462d17d{SSL,[ssl, http/1.1]}{127.0.0.1:45211}
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.e.j.s.Server Started @3608860ms
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45211}
   [junit4]   2> 3592199 ERROR (jetty-launcher-2271-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.2.0
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3592199 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-06T10:25:16.278Z
   [junit4]   2> 3592222 INFO  (zkConnectionManagerCallback-2276-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3592222 INFO  (zkConnectionManagerCallback-2278-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3592223 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3592223 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3592426 INFO  (jetty-launcher-2271-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44942/solr
   [junit4]   2> 3592506 INFO  (zkConnectionManagerCallback-2282-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3592523 INFO  (jetty-launcher-2271-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44942/solr
   [junit4]   2> 3592551 INFO  (zkConnectionManagerCallback-2286-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3592653 INFO  (zkConnectionManagerCallback-2288-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3592710 INFO  (zkConnectionManagerCallback-2290-thread-1-processing-n:127.0.0.1:45211_solr) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3593366 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3593366 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42415_solr
   [junit4]   2> 3593367 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.c.Overseer Overseer (id=99126864514777093-127.0.0.1:42415_solr-n_0000000000) starting
   [junit4]   2> 3593453 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42415_solr
   [junit4]   2> 3593502 INFO  (zkCallback-2287-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3593536 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3593571 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3593572 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45211_solr
   [junit4]   2> 3593608 INFO  (zkCallback-2287-thread-2-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3593650 INFO  (zkCallback-2289-thread-1-processing-n:127.0.0.1:45211_solr) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3593759 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42415.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbb0184
   [junit4]   2> 3593805 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42415.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbb0184
   [junit4]   2> 3593806 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42415.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbb0184
   [junit4]   2> 3593824 INFO  (jetty-launcher-2271-thread-2) [n:127.0.0.1:42415_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_A299E67BDB527C2D-001/tempDir-001/node2/.
   [junit4]   2> 3593829 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45211.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbb0184
   [junit4]   2> 3593866 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45211.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbb0184
   [junit4]   2> 3593866 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45211.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbb0184
   [junit4]   2> 3593867 INFO  (jetty-launcher-2271-thread-1) [n:127.0.0.1:45211_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_A299E67BDB527C2D-001/tempDir-001/node1/.
   [junit4]   2> 3594144 INFO  (zkConnectionManagerCallback-2295-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3594225 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1602b5bab1a0007, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3594290 INFO  (zkConnectionManagerCallback-2299-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3594343 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3594430 INFO  (SUITE-AutoScalingHandlerTest-seed#[A299E67BDB527C2D]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3594492 INFO  (TEST-AutoScalingHandlerTest.testPolicyAndPreferences-seed#[A299E67BDB527C2D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPolicyAndPreferences
   [junit4]   2> 3594665 ERROR (qtp914198665-13363) [n:127.0.0.1:45211_solr    ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-policy={xyz=[{replica=<2, shard=#EACH, node=#ANY}, {nodeRole=!overseer, replica=0}], policy1=[{cores=<2, node=#ANY}, {replica=<2, shard=#EACH, node=#ANY}]}, errorMessages=[cores is only allowed in 'cluster-policy']}],
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:214)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:145)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:311)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 3594666 INFO  (qtp914198665-13363) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=400 QTime=28
   [junit4]   2> 3594762 INFO  (zkConnectionManagerCallback-2303-thread-1-processing-n:127.0.0.1:45211_solr) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3594827 INFO  (qtp914198665-13364) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3594916 INFO  (qtp914198665-13364) [n:127.0.0.1:45211_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3595026 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=12
   [junit4]   2> 3595065 INFO  (qtp914198665-13361) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3595119 INFO  (qtp914198665-13364) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=438
   [junit4]   2> 3595164 INFO  (zkConnectionManagerCallback-2307-thread-1-processing-n:127.0.0.1:45211_solr) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3595176 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3595209 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3595211 INFO  (qtp606878804-13349) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3595212 INFO  (qtp914198665-13361) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3595274 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=142
   [junit4]   2> 3595473 INFO  (zkConnectionManagerCallback-2311-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3595474 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3595475 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3595483 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3595520 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3595583 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 3595666 INFO  (zkConnectionManagerCallback-2315-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3595729 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3595802 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3595805 INFO  (qtp606878804-13349) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3595806 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3595852 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=247
   [junit4]   2> 3595891 INFO  (zkConnectionManagerCallback-2319-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3595905 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3595906 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3595906 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=os.systemLoadAverage&wt=javabin&version=2&group=solr.jvm} status=0 QTime=0
   [junit4]   2> 3595907 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=os.systemLoadAverage&wt=javabin&version=2&group=solr.jvm} status=0 QTime=0
   [junit4]   2> 3595957 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=104
   [junit4]   2> 3596067 INFO  (zkConnectionManagerCallback-2323-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3596116 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3596116 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3596118 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName,os.systemLoadAverage&wt=javabin&version=2&group=solr.jvm,solr.core} status=0 QTime=0
   [junit4]   2> 3596119 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName,os.systemLoadAverage&wt=javabin&version=2&group=solr.jvm,solr.core} status=0 QTime=0
   [junit4]   2> 3596160 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=182
   [junit4]   2> 3596176 INFO  (TEST-AutoScalingHandlerTest.testPolicyAndPreferences-seed#[A299E67BDB527C2D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPolicyAndPreferences
   [junit4]   2> 3596380 INFO  (TEST-AutoScalingHandlerTest.test-seed#[A299E67BDB527C2D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 3596648 INFO  (zkConnectionManagerCallback-2327-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3596649 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3596650 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3596650 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3596664 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3596666 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=241
   [junit4]   2> 3596820 INFO  (zkConnectionManagerCallback-2331-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3596831 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3596832 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3596833 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3596833 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3596868 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=146
   [junit4]   2> 3596945 INFO  (zkConnectionManagerCallback-2335-thread-1-processing-n:127.0.0.1:45211_solr) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3596966 INFO  (qtp914198665-13363) [n:127.0.0.1:45211_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3596966 INFO  (qtp914198665-13363) [n:127.0.0.1:45211_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3596978 INFO  (qtp606878804-13352) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3596978 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597003 INFO  (qtp914198665-13363) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=133
   [junit4]   2> 3597005 ERROR (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{remove-trigger={name=node_lost_trigger}, errorMessages=[Cannot remove trigger: node_lost_trigger because it has active listeners: [xyz, node_lost_trigger.system]]}],
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:214)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:145)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:311)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 3597005 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=400 QTime=1
   [junit4]   2> 3597104 INFO  (zkConnectionManagerCallback-2339-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3597120 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3597121 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3597122 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597123 INFO  (qtp914198665-13358) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597173 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=133
   [junit4]   2> 3597250 INFO  (zkConnectionManagerCallback-2343-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3597267 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3597267 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3597268 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597269 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597282 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1602b5bab1a0013, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3597303 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=129
   [junit4]   2> 3597305 ERROR (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-listener={name=xyz, trigger=node_lost_trigger, stage=[STARTED, ABORTED, SUCCEEDED], beforeAction=execute_plan, class=org.apache.solr.cloud.autoscaling.AutoScaling$HttpTriggerListener, url=http://xyz.com/on_node_lost?node={$LOST_NODE_NAME}}, errorMessages=[A trigger with the name node_lost_trigger does not exist]}],
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:214)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:145)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 3597305 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=400 QTime=1
   [junit4]   2> 3597305 INFO  (TEST-AutoScalingHandlerTest.test-seed#[A299E67BDB527C2D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> 3597449 INFO  (TEST-AutoScalingHandlerTest.testReadApi-seed#[A299E67BDB527C2D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testReadApi
   [junit4]   2> 3597538 INFO  (zkConnectionManagerCallback-2347-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3597543 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3597544 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3597544 INFO  (qtp606878804-13349) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597546 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597595 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=144
   [junit4]   2> 3597620 INFO  (zkConnectionManagerCallback-2351-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3597620 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3597621 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3597621 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597622 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 3597649 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=52
   [junit4]   2> 3597717 INFO  (zkConnectionManagerCallback-2355-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3597718 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3597718 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3597720 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3597721 INFO  (qtp914198665-13365) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3597810 INFO  (qtp606878804-13356) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=161
   [junit4]   2> 3597864 INFO  (zkConnectionManagerCallback-2359-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3597937 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3598010 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3598031 INFO  (qtp606878804-13350) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3598047 INFO  (qtp914198665-13363) [n:127.0.0.1:45211_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3598068 INFO  (qtp606878804-13355) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=256
   [junit4]   2> 3598070 INFO  (qtp606878804-13353) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 3598123 INFO  (zkConnectionManagerCallback-2363-thread-1-processing-n:127.0.0.1:42415_solr) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3598124 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3598125 INFO  (qtp606878804-13354) [n:127.0.0.1:42415_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44942/solr ready
   [junit4]   2> 3598126 INFO  (qtp606878804-13349) [n:127.0.0.1:42415_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=so

[...truncated too long message...]

esource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java (at line 578)
 [ecj-lint] tvWindowStream = new OffsetWindowTokenFilter(tvStream);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tvWindowStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/request/SimpleFacets.java (at line 943)
 [ecj-lint] fastForRandomSet = new HashDocSet(sset.getDocs(), 0, sset.size());
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fastForRandomSet' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java (at line 33)
 [ecj-lint] new SmileWriter(out, request, response).writeResponse();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/schema/OpenExchangeRatesOrgProvider.java (at line 146)
 [ecj-lint] ratesJsonStream = resourceLoader.openResource(ratesFileLocation);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ratesJsonStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java (at line 1315)
 [ecj-lint] UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), collapseField), mapping);
 [ecj-lint]                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 145)
 [ecj-lint] HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint]           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 154)
 [ecj-lint] HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint]           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 163)
 [ecj-lint] HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint]           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 16. ERROR in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/facet/UniqueSinglevaluedSlotAcc.java (at line 30)
 [ecj-lint] import org.apache.solr.search.SolrIndexSearcher;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.search.SolrIndexSearcher is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 17. ERROR in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/facet/UniqueSlotAcc.java (at line 24)
 [ecj-lint] import org.apache.lucene.index.LeafReaderContext;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.lucene.index.LeafReaderContext is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/function/OrdFieldSource.java (at line 79)
 [ecj-lint] SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint]                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/search/function/ReverseOrdFieldSource.java (at line 79)
 [ecj-lint] SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint]                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (at line 291)
 [ecj-lint] throw new SolrException(ErrorCode.SERVER_ERROR, "Error occurred while loading solr.xml from zookeeper", e);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'loader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/CdcrTransactionLog.java (at line 101)
 [ecj-lint] ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint]                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 191)
 [ecj-lint] LogCodec codec = new LogCodec(resolver);
 [ecj-lint]         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 261)
 [ecj-lint] LogCodec codec = new LogCodec(resolver);
 [ecj-lint]         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint]                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint]                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 260)
 [ecj-lint] LogCodec codec = new LogCodec(resolver);
 [ecj-lint]         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 275)
 [ecj-lint] LogCodec codec = new LogCodec(resolver);
 [ecj-lint]         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 529)
 [ecj-lint] LogCodec codec = new LogCodec(resolver);
 [ecj-lint]         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] BAOS bos =  new BAOS();
 [ecj-lint]     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] BAOS bos =  new BAOS();
 [ecj-lint]     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 31. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/java/org/apache/solr/util/SolrLogLayout.java (at line 149)
 [ecj-lint] SolrQueryRequest req = requestInfo == null ? null : requestInfo.getReq();
 [ecj-lint]                 ^^^
 [ecj-lint] Resource leak: 'req' is never closed
 [ecj-lint] ----------
 [ecj-lint] 31 problems (2 errors, 29 warnings)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build.xml:684: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2105: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2144: Compile failed; see the compiler error output for details.

Total time: 303 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 273 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/273/

All tests passed

Build Log:
[...truncated 1065 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4044992831442732580301.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] #
   [junit4] # There is insufficient memory for the Java Runtime Environment to continue.
   [junit4] # Native memory allocation (mmap) failed to map 1572864 bytes for committing reserved memory.
   [junit4] # An error report file with more information is saved as:
   [junit4] # /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/J1/hs_err_pid7234.log
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4044736043498099302961.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000ffd00000, 1572864, 0) failed; error='Cannot allocate memory' (errno=12)
   [junit4] <<< JVM J1: EOF ----

[...truncated 705 lines...]
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_144/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=5AF8949AA6AFADAF -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.3.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=7.3.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x -Djava.security.egd=file:/dev/./urandom -Djunit4.childvm.cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.3.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4047567217770388814110.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4043975436851533545118.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4044992831442732580301.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4044736043498099302961.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:779: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:1512: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:1038: At least one slave process threw an exception, first: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4044992831442732580301.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J1-20171206_192701_4044736043498099302961.syserr

Total time: 55 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-7.x #266 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 274 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/274/

13 tests failed.
FAILED:  org.apache.solr.cloud.ShardSplitTest.test

Error Message:
Error from server at <a href="http://127.0.0.1:40473:">http://127.0.0.1:40473: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at <a href="http://127.0.0.1:40473:">http://127.0.0.1:40473: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
        at __randomizedtesting.SeedInfo.seed([9BBE8EC552063651:13EAB11FFCFA5BA9]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:417)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:333)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest

Error Message:
42 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:     1) Thread[id=19529, name=qtp2035310209-19529, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=19538, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=19614, name=ScheduledTrigger-8094-thread-2, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=19602, name=OverseerCollectionConfigSetProcessor-99132238547582989-127.0.0.1:43374_-n_0000000003, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:387)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=19604, name=OverseerAutoScalingTriggerThread-99132238547582989-127.0.0.1:43374_-n_0000000003, state=WAITING, group=Overseer autoscaling triggers]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=19530, name=qtp2035310209-19530, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=19531, name=qtp2035310209-19531, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=19563, name=qtp335278672-19563, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=19618, name=ScheduledTrigger-8094-thread-4, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   10) Thread[id=19546, name=Thread-3589, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   11) Thread[id=19568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   12) Thread[id=19577, name=Thread-3598, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   13) Thread[id=19525, name=qtp2035310209-19525, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=19564, name=org.eclipse.jetty.server.session.HashSessionManager@69d579dfTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=19569, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=19841, name=AutoscalingActionExecutor-8095-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   17) Thread[id=19544, name=zkConnectionManagerCallback-4420-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=19553, name=searcherExecutor-8082-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=19528, name=qtp2035310209-19528, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   20) Thread[id=19593, name=zkConnectionManagerCallback-4420-thread-1-processing-n:127.0.0.1:43374_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)   21) Thread[id=19596, name=zkConnectionManagerCallback-4429-thread-1-processing-n:127.0.0.1:46642_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)   22) Thread[id=19609, name=ScheduledTrigger-8094-thread-1, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   23) Thread[id=19556, name=qtp335278672-19556, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   24) Thread[id=19560, name=qtp335278672-19560, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   25) Thread[id=19615, name=ScheduledTrigger-8094-thread-3, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   26) Thread[id=19559, name=qtp335278672-19559, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   27) Thread[id=19575, name=zkConnectionManagerCallback-4429-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   28) Thread[id=19550, name=Scheduler-1966807769, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   29) Thread[id=19527, name=qtp2035310209-19527-acceptor-0@4c2be8c4-ServerConnector@1065f7ab{HTTP/1.1,[http/1.1]}{127.0.0.1:43374}, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   30) Thread[id=19555, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   31) Thread[id=19594, name=zkConnectionManagerCallback-4420-thread-1-processing-n:127.0.0.1:43374_-EventThread, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   32) Thread[id=19603, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   33) Thread[id=19597, name=zkConnectionManagerCallback-4429-thread-1-processing-n:127.0.0.1:46642_-EventThread, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   34) Thread[id=19532, name=qtp2035310209-19532, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   35) Thread[id=19533, name=org.eclipse.jetty.server.session.HashSessionManager@ec6969aTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   36) Thread[id=19601, name=OverseerStateUpdate-99132238547582989-127.0.0.1:43374_-n_0000000003, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:435)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:203)         at java.lang.Thread.run(Thread.java:748)   37) Thread[id=19526, name=qtp2035310209-19526, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   38) Thread[id=19557, name=qtp335278672-19557, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   39) Thread[id=19558, name=qtp335278672-19558-acceptor-0@2a96b2a6-ServerConnector@2ea12d1{HTTP/1.1,[http/1.1]}{127.0.0.1:46642}, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   40) Thread[id=19537, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   41) Thread[id=19562, name=qtp335278672-19562, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)   42) Thread[id=19561, name=qtp335278672-19561, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 42 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
   1) Thread[id=19529, name=qtp2035310209-19529, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=19538, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=19614, name=ScheduledTrigger-8094-thread-2, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=19602, name=OverseerCollectionConfigSetProcessor-99132238547582989-127.0.0.1:43374_-n_0000000003, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:387)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=19604, name=OverseerAutoScalingTriggerThread-99132238547582989-127.0.0.1:43374_-n_0000000003, state=WAITING, group=Overseer autoscaling triggers]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=19530, name=qtp2035310209-19530, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=19531, name=qtp2035310209-19531, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=19563, name=qtp335278672-19563, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
   9) Thread[id=19618, name=ScheduledTrigger-8094-thread-4, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  10) Thread[id=19546, name=Thread-3589, state=WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)
  11) Thread[id=19568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  12) Thread[id=19577, name=Thread-3598, state=WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)
  13) Thread[id=19525, name=qtp2035310209-19525, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  14) Thread[id=19564, name=org.eclipse.jetty.server.session.HashSessionManager@69d579dfTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  15) Thread[id=19569, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  16) Thread[id=19841, name=AutoscalingActionExecutor-8095-thread-1, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  17) Thread[id=19544, name=zkConnectionManagerCallback-4420-thread-1, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  18) Thread[id=19553, name=searcherExecutor-8082-thread-1, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  19) Thread[id=19528, name=qtp2035310209-19528, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  20) Thread[id=19593, name=zkConnectionManagerCallback-4420-thread-1-processing-n:127.0.0.1:43374_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
  21) Thread[id=19596, name=zkConnectionManagerCallback-4429-thread-1-processing-n:127.0.0.1:46642_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
  22) Thread[id=19609, name=ScheduledTrigger-8094-thread-1, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  23) Thread[id=19556, name=qtp335278672-19556, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  24) Thread[id=19560, name=qtp335278672-19560, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  25) Thread[id=19615, name=ScheduledTrigger-8094-thread-3, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  26) Thread[id=19559, name=qtp335278672-19559, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  27) Thread[id=19575, name=zkConnectionManagerCallback-4429-thread-1, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  28) Thread[id=19550, name=Scheduler-1966807769, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  29) Thread[id=19527, name=qtp2035310209-19527-acceptor-0@4c2be8c4-ServerConnector@1065f7ab{HTTP/1.1,[http/1.1]}{127.0.0.1:43374}, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  30) Thread[id=19555, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  31) Thread[id=19594, name=zkConnectionManagerCallback-4420-thread-1-processing-n:127.0.0.1:43374_-EventThread, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
  32) Thread[id=19603, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  33) Thread[id=19597, name=zkConnectionManagerCallback-4429-thread-1-processing-n:127.0.0.1:46642_-EventThread, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
  34) Thread[id=19532, name=qtp2035310209-19532, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  35) Thread[id=19533, name=org.eclipse.jetty.server.session.HashSessionManager@ec6969aTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
  36) Thread[id=19601, name=OverseerStateUpdate-99132238547582989-127.0.0.1:43374_-n_0000000003, state=TIMED_WAITING, group=Overseer state updater.]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:435)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:203)
        at java.lang.Thread.run(Thread.java:748)
  37) Thread[id=19526, name=qtp2035310209-19526, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  38) Thread[id=19557, name=qtp335278672-19557, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  39) Thread[id=19558, name=qtp335278672-19558-acceptor-0@2a96b2a6-ServerConnector@2ea12d1{HTTP/1.1,[http/1.1]}{127.0.0.1:46642}, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
  40) Thread[id=19537, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  41) Thread[id=19562, name=qtp335278672-19562, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
  42) Thread[id=19561, name=qtp335278672-19561, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([9BBE8EC552063651]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=19529, name=qtp2035310209-19529, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=19614, name=ScheduledTrigger-8094-thread-2, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=19593, name=zkConnectionManagerCallback-4420-thread-1-processing-n:127.0.0.1:43374_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) Thread[id=19596, name=zkConnectionManagerCallback-4429-thread-1-processing-n:127.0.0.1:46642_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170)    5) Thread[id=19609, name=ScheduledTrigger-8094-thread-1, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:315)         at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:57)         at org.apache.solr.cloud.autoscaling.TriggerBase.saveState(TriggerBase.java:203)         at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.run(ScheduledTriggers.java:554)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=19618, name=ScheduledTrigger-8094-thread-4, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=19556, name=qtp335278672-19556, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=19615, name=ScheduledTrigger-8094-thread-3, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    9) Thread[id=19546, name=Thread-3589, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   10) Thread[id=19575, name=zkConnectionManagerCallback-4429-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   11) Thread[id=19550, name=Scheduler-1966807769, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   12) Thread[id=19577, name=Thread-3598, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1717)   13) Thread[id=19564, name=org.eclipse.jetty.server.session.HashSessionManager@69d579dfTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   14) Thread[id=19841, name=AutoscalingActionExecutor-8095-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   15) Thread[id=19533, name=org.eclipse.jetty.server.session.HashSessionManager@ec6969aTimer, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   16) Thread[id=19526, name=qtp2035310209-19526, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   17) Thread[id=19557, name=qtp335278672-19557, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.IOUtil.drain(Native Method)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:748)   18) Thread[id=19544, name=zkConnectionManagerCallback-4420-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)   19) Thread[id=19553, name=searcherExecutor-8082-thread-1, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=19529, name=qtp2035310209-19529, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=19614, name=ScheduledTrigger-8094-thread-2, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=19593, name=zkConnectionManagerCallback-4420-thread-1-processing-n:127.0.0.1:43374_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=19596, name=zkConnectionManagerCallback-4429-thread-1-processing-n:127.0.0.1:46642_-SendThread(127.0.0.1:49887), state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170)
   5) Thread[id=19609, name=ScheduledTrigger-8094-thread-1, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:315)
        at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:57)
        at org.apache.solr.cloud.autoscaling.TriggerBase.saveState(TriggerBase.java:203)
        at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.run(ScheduledTriggers.java:554)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=19618, name=ScheduledTrigger-8094-thread-4, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=19556, name=qtp335278672-19556, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelec

[...truncated too long message...]

main = solr.core.foo.shard1.replica_n4, service url = null, agent id = null] for registry solr.core.foo.shard1.replica_n4 / com.codahale.metrics.MetricRegistry@6b5a9e01
   [junit4]   2> 613142 INFO  (coreCloseExecutor-573-thread-6) [n:127.0.0.1:51661_solr c:localShardsTestColl s:shard1 r:core_node4 x:localShardsTestColl_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=416861671
   [junit4]   2> 613613 INFO  (coreCloseExecutor-575-thread-4) [n:127.0.0.1:44932_solr c:localShardsTestColl s:shard1 r:core_node6 x:localShardsTestColl_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@30c9351: rootName = solr_44932, domain = solr.core.localShardsTestColl.shard1.replica_n5, service url = null, agent id = null] for registry solr.core.localShardsTestColl.shard1.replica_n5 / com.codahale.metrics.MetricRegistry@7542601c
   [junit4]   2> 613717 INFO  (coreCloseExecutor-575-thread-5) [n:127.0.0.1:44932_solr c:localShardsTestColl s:shard2 r:core_node12 x:localShardsTestColl_shard2_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.localShardsTestColl.shard2.replica_n11, tag=891366563
   [junit4]   2> 613769 INFO  (coreCloseExecutor-574-thread-7) [n:127.0.0.1:33400_solr c:foo s:shard1 r:core_node6 x:foo_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.foo.shard1.leader, tag=871895232
   [junit4]   2> 613772 INFO  (coreCloseExecutor-574-thread-8) [n:127.0.0.1:33400_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.overwrite.shard1.replica_n1, tag=1469459194
   [junit4]   2> 613772 INFO  (coreCloseExecutor-574-thread-8) [n:127.0.0.1:33400_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@34d3edbc: rootName = solr_33400, domain = solr.core.overwrite.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.overwrite.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@1ded0c2e
   [junit4]   2> 613788 INFO  (coreCloseExecutor-575-thread-5) [n:127.0.0.1:44932_solr c:localShardsTestColl s:shard2 r:core_node12 x:localShardsTestColl_shard2_replica_n11] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3c155fb2: rootName = solr_44932, domain = solr.core.localShardsTestColl.shard2.replica_n11, service url = null, agent id = null] for registry solr.core.localShardsTestColl.shard2.replica_n11 / com.codahale.metrics.MetricRegistry@14c340a4
   [junit4]   2> 613881 INFO  (coreCloseExecutor-574-thread-8) [n:127.0.0.1:33400_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.overwrite.shard1.leader, tag=1469459194
   [junit4]   2> 613882 INFO  (coreCloseExecutor-575-thread-5) [n:127.0.0.1:44932_solr c:localShardsTestColl s:shard2 r:core_node12 x:localShardsTestColl_shard2_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard2.leader, tag=891366563
   [junit4]   2> 613882 INFO  (coreCloseExecutor-574-thread-1) [n:127.0.0.1:33400_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=1756682062
   [junit4]   2> 613891 INFO  (coreCloseExecutor-575-thread-6) [n:127.0.0.1:44932_solr c:foo s:shard1 r:core_node5 x:foo_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.foo.shard1.replica_n2, tag=714307742
   [junit4]   2> 613892 INFO  (coreCloseExecutor-575-thread-6) [n:127.0.0.1:44932_solr c:foo s:shard1 r:core_node5 x:foo_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7616d78b: rootName = solr_44932, domain = solr.core.foo.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.foo.shard1.replica_n2 / com.codahale.metrics.MetricRegistry@2703497a
   [junit4]   2> 613908 INFO  (coreCloseExecutor-574-thread-2) [n:127.0.0.1:33400_solr c:multicollection1 s:shard2 r:core_node4 x:multicollection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection1.shard2.leader, tag=102852045
   [junit4]   2> 613909 INFO  (coreCloseExecutor-574-thread-4) [n:127.0.0.1:33400_solr c:localShardsTestColl s:shard3 r:core_node14 x:localShardsTestColl_shard3_replica_n13] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=179863590
   [junit4]   2> 613926 INFO  (coreCloseExecutor-575-thread-6) [n:127.0.0.1:44932_solr c:foo s:shard1 r:core_node5 x:foo_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.foo.shard1.leader, tag=714307742
   [junit4]   2> 613948 INFO  (coreCloseExecutor-575-thread-4) [n:127.0.0.1:44932_solr c:localShardsTestColl s:shard1 r:core_node6 x:localShardsTestColl_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=1751630916
   [junit4]   2> 614481 INFO  (jetty-closer-93-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@49842d3b{/solr,null,UNAVAILABLE}
   [junit4]   2> 614550 INFO  (jetty-closer-93-thread-2) [    ] o.a.s.c.Overseer Overseer (id=99132304772038670-127.0.0.1:51661_solr-n_0000000003) closing
   [junit4]   2> 614550 INFO  (OverseerStateUpdate-99132304772038670-127.0.0.1:51661_solr-n_0000000003) [n:127.0.0.1:51661_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:51661_solr
   [junit4]   2> 614556 WARN  (OverseerAutoScalingTriggerThread-99132304772038670-127.0.0.1:51661_solr-n_0000000003) [n:127.0.0.1:51661_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 614581 INFO  (jetty-closer-93-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@bc0f75c{/solr,null,UNAVAILABLE}
   [junit4]   2> 614596 INFO  (jetty-closer-93-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5fa1358e{/solr,null,UNAVAILABLE}
   [junit4]   2> 614616 ERROR (SUITE-CloudSolrClientTest-seed#[E3B3AC6C6A4A77D]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 614616 INFO  (SUITE-CloudSolrClientTest-seed#[E3B3AC6C6A4A77D]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:56653 56653
   [junit4]   2> 616420 INFO  (Thread-126) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:56653 56653
   [junit4]   2> 616421 WARN  (Thread-126) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 13 /solr/aliases.json
   [junit4]   2> 6 /solr/security.json
   [junit4]   2> 6 /solr/configs/conf
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 39 /solr/collections/localShardsTestColl/state.json
   [junit4]   2> 27 /solr/collections/foo/state.json
   [junit4]   2> 18 /solr/collections/2nd_collection/state.json
   [junit4]   2> 17 /solr/collections/multicollection1/state.json
   [junit4]   2> 16 /solr/collections/stale_state_test_col/state.json
   [junit4]   2> 15 /solr/collections/collection1/state.json
   [junit4]   2> 15 /solr/collections/nemesis/state.json
   [junit4]   2> 13 /solr/clusterprops.json
   [junit4]   2> 13 /solr/clusterstate.json
   [junit4]   2> 12 /solr/collections/multicollection2/state.json
   [junit4]   2> 6 /solr/collections/overwrite/state.json
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 13 /solr/collections
   [junit4]   2> 10 /solr/live_nodes
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_E3B3AC6C6A4A77D-001
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=6, maxDocsPerChunk=7, blockSize=668), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=6, blockSize=668)), sim=RandomSimilarity(queryNorm=true): {}, locale=ar-BH, timezone=Europe/Gibraltar
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=112319256,total=458752000
   [junit4]   2> NOTE: All tests run in this JVM: [SolrExampleXMLTest, CosineEvaluatorTest, SolrExampleBinaryTest, LargeVolumeJettyTest, CommonParamsTest, TestValidatingJsonMap, GetByIdTest, ExclusiveOrEvaluatorTest, UuidEvaluatorTest, MultiplyEvaluatorTest, GreaterThanEqualToEvaluatorTest, TestBatchUpdate, HyperbolicSineEvaluatorTest, SelectWithEvaluatorsTest, JdbcDriverTest, NormalizeEvaluatorTest, TestCoreAdmin, TestSuggesterResponse, DivideEvaluatorTest, JettyWebappTest, TestCollectionAdminRequest, TestHash, TestJavaBinCodec, SolrExampleJettyTest, CloudSolrClientTest]
   [junit4] Completed [106/146 (1!)] on J2 in 166.97s, 16 tests, 5 errors <<< FAILURES!

[...truncated 46568 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 275 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/275/

11 tests failed.
FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddShard

Error Message:
Could not load collection from ZK: TooManyReplicasWhenAddingShards

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: TooManyReplicasWhenAddingShards
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:EF68157CD013E539]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:110)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
        at org.apache.solr.cloud.CollectionTooManyReplicasTest.getAllNodeNames(CollectionTooManyReplicasTest.java:217)
        at org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddShard(CollectionTooManyReplicasTest.java:146)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/TooManyReplicasWhenAddingShards/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 45 more


FAILED:  org.apache.solr.cloud.DistribJoinFromCollectionTest.testNoScore

Error Message:
Error from server at https://127.0.0.1:34182/solr/to_2x2: SolrCloud join: Collection 'from_1x4Alias' not found!

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34182/solr/to_2x2: SolrCloud join: Collection 'from_1x4Alias' not found!
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:74A27F6F764A243E]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.DistribJoinFromCollectionTest.testJoins(DistribJoinFromCollectionTest.java:173)
        at org.apache.solr.cloud.DistribJoinFromCollectionTest.testNoScore(DistribJoinFromCollectionTest.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:43794C0726FA568A]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:112)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:3260E072B47376F8]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:279)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenRenew

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:FCB687C3B0CAE6D6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.renewDelegationToken(TestSolrCloudWithDelegationTokens.java:130)
        at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.verifyDelegationTokenRenew(TestSolrCloudWithDelegationTokens.java:316)
        at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenRenew(TestSolrCloudWithDelegationTokens.java:333)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:A39BA07F1DE23AEC]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync(ZkSolrClientTest.java:252)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:9C04886853F4D969]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi(AutoScalingHandlerTest.java:724)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionSplitShard

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:53075/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:53075/solr
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:1EA30026384CBACC]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionSplitShard(TestPolicyCloud.java:173)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.TestPullReplica.testAddDocs

Error Message:
Could not load collection from ZK: pull_replica_test_add_docs

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: pull_replica_test_add_docs
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
        at org.apache.solr.cloud.TestPullReplica.tearDown(TestPullReplica.java:112)
        at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/pull_replica_test_add_docs/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 39 more


FAILED:  org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir

Error Message:
Could not load collection from ZK: cdcr-cluster1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: cdcr-cluster1
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:8EF6833F90287730]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.cdcr.CdcrTestsUtil.invokeCdcrAction(CdcrTestsUtil.java:64)
        at org.apache.solr.cloud.cdcr.CdcrTestsUtil.cdcrStart(CdcrTestsUtil.java:41)
        at org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir(CdcrBidirectionalTest.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/cdcr-cluster1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 49 more


FAILED:  org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory

Error Message:
expected:<8> but was:<10>

Stack Trace:
java.lang.AssertionError: expected:<8> but was:<10>
        at __randomizedtesting.SeedInfo.seed([CB2D73DD88063B72:A6D1D720324EC475]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:259)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11892 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestPullReplica
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/init-core-data-001
   [junit4]   2> 1343418 WARN  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 1343418 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1343419 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1343420 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1343456 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001
   [junit4]   2> 1343456 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1343456 INFO  (Thread-256) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1343456 INFO  (Thread-256) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1343524 ERROR (Thread-256) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1343573 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40878
   [junit4]   2> 1343707 INFO  (zkConnectionManagerCallback-556-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1345513 INFO  (jetty-launcher-553-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1345514 INFO  (jetty-launcher-553-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1345515 INFO  (jetty-launcher-553-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@34599a40{/solr,null,AVAILABLE}
   [junit4]   2> 1345516 INFO  (jetty-launcher-553-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3815652b{SSL,[ssl, http/1.1]}{127.0.0.1:53474}
   [junit4]   2> 1345516 INFO  (jetty-launcher-553-thread-2) [    ] o.e.j.s.Server Started @1367467ms
   [junit4]   2> 1345516 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53474}
   [junit4]   2> 1345517 ERROR (jetty-launcher-553-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1345517 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 1345517 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1345517 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1345517 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T01:21:58.565Z
   [junit4]   2> 1346348 INFO  (jetty-launcher-553-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@48201166{/solr,null,AVAILABLE}
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1092f6ad{SSL,[ssl, http/1.1]}{127.0.0.1:60510}
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.e.j.s.Server Started @1368300ms
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60510}
   [junit4]   2> 1346349 ERROR (jetty-launcher-553-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1346349 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T01:21:59.397Z
   [junit4]   2> 1348012 INFO  (zkConnectionManagerCallback-558-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1348024 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1348029 INFO  (zkConnectionManagerCallback-560-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1348064 INFO  (jetty-launcher-553-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40878/solr
   [junit4]   2> 1348137 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1348185 INFO  (jetty-launcher-553-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40878/solr
   [junit4]   2> 1348188 INFO  (zkConnectionManagerCallback-564-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1348212 INFO  (zkConnectionManagerCallback-568-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1348394 INFO  (zkConnectionManagerCallback-570-thread-1-processing-n:127.0.0.1:53474_solr) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1348561 INFO  (zkConnectionManagerCallback-572-thread-1-processing-n:127.0.0.1:60510_solr) [n:127.0.0.1:60510_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1348579 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1348596 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60510_solr
   [junit4]   2> 1348597 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.c.Overseer Overseer (id=99136052691337222-127.0.0.1:60510_solr-n_0000000000) starting
   [junit4]   2> 1348676 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60510_solr
   [junit4]   2> 1348693 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1349224 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60510.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1349247 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60510.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1349247 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60510.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1349280 INFO  (jetty-launcher-553-thread-1) [n:127.0.0.1:60510_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node1/.
   [junit4]   2> 1349498 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1349499 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1349528 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53474_solr
   [junit4]   2> 1349806 INFO  (zkCallback-569-thread-1-processing-n:127.0.0.1:53474_solr) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1349880 INFO  (zkCallback-571-thread-2-processing-n:127.0.0.1:60510_solr) [n:127.0.0.1:60510_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1350603 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1350706 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1351622 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1351624 INFO  (jetty-launcher-553-thread-2) [n:127.0.0.1:53474_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/.
   [junit4]   2> 1353621 INFO  (zkConnectionManagerCallback-577-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1353665 INFO  (zkConnectionManagerCallback-581-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1353666 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1353666 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40878/solr ready
   [junit4]   2> 1353668 INFO  (SUITE-TestPullReplica-seed#[CB2D73DD88063B72]-worker) [    ] o.a.s.c.TestPullReplica Using legacyCloud?: false
   [junit4]   2> 1354503 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1354507 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1354611 INFO  (TEST-TestPullReplica.testCreateDelete-seed#[CB2D73DD88063B72]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateDelete {seed=[CB2D73DD88063B72:D03D82917EF6C342]}
   [junit4]   2> 1354832 INFO  (OverseerThreadFactory-622-thread-1-processing-n:127.0.0.1:60510_solr) [n:127.0.0.1:60510_solr    ] o.a.s.c.CreateCollectionCmd Create collection pull_replica_test_create_delete
   [junit4]   2> 1354833 WARN  (OverseerThreadFactory-622-thread-1-processing-n:127.0.0.1:60510_solr) [n:127.0.0.1:60510_solr    ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 4 on collection pull_replica_test_create_delete is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 1354951 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:60510/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1354963 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard1_replica_p3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:53474/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1354984 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard1_replica_p5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:60510/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1355003 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard1_replica_p6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:53474/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1355021 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard2_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:60510/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1355043 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard2_replica_p10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:53474/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1355052 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard2_replica_p12",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:60510/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1355064 INFO  (OverseerStateUpdate-99136052691337222-127.0.0.1:60510_solr-n_0000000000) [n:127.0.0.1:60510_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_create_delete",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"pull_replica_test_create_delete_shard2_replica_p14",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:53474/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1357014 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node13&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard2_replica_p10&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1357014 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1357028 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node16&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard2_replica_p14&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1357029 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node9&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard1_replica_p6&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1357048 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard1_replica_p3&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1357221 INFO  (zkCallback-569-thread-1-processing-n:127.0.0.1:53474_solr) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_create_delete/state.json] for collection [pull_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1358169 INFO  (zkCallback-569-thread-1-processing-n:127.0.0.1:53474_solr) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_create_delete/state.json] for collection [pull_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1358523 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 1358563 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 1358598 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 1358602 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.s.IndexSchema [pull_replica_test_create_delete_shard2_replica_p14] Schema name=minimal
   [junit4]   2> 1358742 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.s.IndexSchema [pull_replica_test_create_delete_shard1_replica_p6] Schema name=minimal
   [junit4]   2> 1358760 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1358760 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_create_delete_shard1_replica_p6' using configuration from collection pull_replica_test_create_delete, trusted=true
   [junit4]   2> 1358793 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.s.IndexSchema [pull_replica_test_create_delete_shard1_replica_p3] Schema name=minimal
   [junit4]   2> 1358795 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1358795 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_create_delete_shard1_replica_p3' using configuration from collection pull_replica_test_create_delete, trusted=true
   [junit4]   2> 1358799 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1358799 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_create_delete_shard2_replica_p14' using configuration from collection pull_replica_test_create_delete, trusted=true
   [junit4]   2> 1359066 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 1359110 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.s.IndexSchema [pull_replica_test_create_delete_shard2_replica_p10] Schema name=minimal
   [junit4]   2> 1360351 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.core.pull_replica_test_create_delete.shard1.replica_p6' (registry 'solr.core.pull_replica_test_create_delete.shard1.replica_p6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1360351 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.core.pull_replica_test_create_delete.shard1.replica_p3' (registry 'solr.core.pull_replica_test_create_delete.shard1.replica_p3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1360351 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1360351 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1360351 INFO  (qtp280044451-1908) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node4 x:pull_replica_test_create_delete_shard1_replica_p3] o.a.s.c.SolrCore [[pull_replica_test_create_delete_shard1_replica_p3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/pull_replica_test_create_delete_shard1_replica_p3], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/./pull_replica_test_create_delete_shard1_replica_p3/data/]
   [junit4]   2> 1360351 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.c.SolrCore [[pull_replica_test_create_delete_shard1_replica_p6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/pull_replica_test_create_delete_shard1_replica_p6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/./pull_replica_test_create_delete_shard1_replica_p6/data/]
   [junit4]   2> 1360372 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.core.pull_replica_test_create_delete.shard2.replica_p14' (registry 'solr.core.pull_replica_test_create_delete.shard2.replica_p14') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1360372 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1360372 INFO  (qtp280044451-1905) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node16 x:pull_replica_test_create_delete_shard2_replica_p14] o.a.s.c.SolrCore [[pull_replica_test_create_delete_shard2_replica_p14] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/pull_replica_test_create_delete_shard2_replica_p14], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/./pull_replica_test_create_delete_shard2_replica_p14/data/]
   [junit4]   2> 1360472 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1360472 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_create_delete_shard2_replica_p10' using configuration from collection pull_replica_test_create_delete, trusted=true
   [junit4]   2> 1360473 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53474.solr.core.pull_replica_test_create_delete.shard2.replica_p10' (registry 'solr.core.pull_replica_test_create_delete.shard2.replica_p10') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dbf500e
   [junit4]   2> 1360473 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1360473 INFO  (qtp280044451-1904) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard2 r:core_node13 x:pull_replica_test_create_delete_shard2_replica_p10] o.a.s.c.SolrCore [[pull_replica_test_create_delete_shard2_replica_p10] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/pull_replica_test_create_delete_shard2_replica_p10], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_CB2D73DD88063B72-001/tempDir-001/node2/./pull_replica_test_create_delete_shard2_replica_p10/data/]
   [junit4]   2> 1360502 INFO  (qtp772050368-1918) [n:127.0.0.1:60510_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard1_replica_p5&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1360504 INFO  (qtp772050368-1921) [n:127.0.0.1:60510_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node11&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard2_replica_n8&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1360520 INFO  (qtp772050368-1915) [n:127.0.0.1:60510_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node15&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard2_replica_p12&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1360521 INFO  (qtp772050368-1915) [n:127.0.0.1:60510_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1360601 INFO  (zkCallback-569-thread-1-processing-n:127.0.0.1:53474_solr) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_create_delete/state.json] for collection [pull_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1360602 INFO  (zkCallback-571-thread-2-processing-n:127.0.0.1:60510_solr) [n:127.0.0.1:60510_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_create_delete/state.json] for collection [pull_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1360694 INFO  (qtp772050368-1920) [n:127.0.0.1:60510_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=pull_replica_test_create_delete_shard1_replica_n1&action=CREATE&numShards=2&collection=pull_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1360847 INFO  (zkCallback-571-thread-2-processing-n:127.0.0.1:60510_solr) [n:127.0.0.1:60510_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_create_delete/state.json] for collection [pull_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1360851 INFO  (zkCallback-569-thread-1-processing-n:127.0.0.1:53474_solr) [n:127.0.0.1:53474_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_create_delete/state.json] for collection [pull_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1361120 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1361120 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1361209 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_test_create_delete_shard1_replica_p6] o.a.s.s.SolrIndexSearcher Opening [Searcher@14dd6193[pull_replica_test_create_delete_shard1_replica_p6] main]
   [junit4]   2> 1361211 INFO  (qtp280044451-1910) [n:127.0.0.1:53474_solr c:pull_replica_test_create_delete s:shard1 r:core_node9 x:pull_replica_tes

[...truncated too long message...]

, TestSolrConfigHandler, TestSystemCollAutoCreate, V2ApiIntegrationTest, AutoscalingHistoryHandlerTest]
   [junit4] Completed [594/751 (11!)] on J1 in 90.42s, 1 test, 1 failure <<< FAILURES!

[...truncated 35186 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /tmp/ecj516222754
 [ecj-lint] Compiling 188 source files to /tmp/ecj516222754
 [ecj-lint] ----------
 [ecj-lint] 1. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/analysis/BaseTokenStreamTestCase.java (at line 801)
 [ecj-lint] ts = a.tokenStream("dummy", useCharFilter ? new MockCharFilter(reader, remainder) : reader);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ts' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 2. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/analysis/BaseTokenStreamTestCase.java (at line 837)
 [ecj-lint] reader = new MockReaderWrapper(random, reader);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/analysis/BaseTokenStreamTestCase.java (at line 913)
 [ecj-lint] reader = new MockReaderWrapper(random, reader);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/codecs/mockrandom/MockRandomPostingsFormat.java (at line 260)
 [ecj-lint] throw new AssertionError();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'postingsWriter' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/codecs/mockrandom/MockRandomPostingsFormat.java (at line 393)
 [ecj-lint] throw new AssertionError();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'postingsReader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/geo/BaseGeoPointTestCase.java (at line 1271)
 [ecj-lint] RandomIndexWriter writer = new RandomIndexWriter(random(), dir, iwc);
 [ecj-lint]                  ^^^^^^
 [ecj-lint] Resource leak: 'writer' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BaseCompressingDocValuesFormatTestCase.java (at line 47)
 [ecj-lint] final IndexWriter iwriter = new IndexWriter(dir, iwc);
 [ecj-lint]                  ^^^^^^^
 [ecj-lint] Resource leak: 'iwriter' is never closed
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BaseCompressingDocValuesFormatTestCase.java (at line 81)
 [ecj-lint] final IndexWriter iwriter = new IndexWriter(dir, iwc);
 [ecj-lint]                  ^^^^^^^
 [ecj-lint] Resource leak: 'iwriter' is never closed
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BaseCompressingDocValuesFormatTestCase.java (at line 108)
 [ecj-lint] final IndexWriter iwriter = new IndexWriter(dir, iwc);
 [ecj-lint]                  ^^^^^^^
 [ecj-lint] Resource leak: 'iwriter' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BasePointsFormatTestCase.java (at line 622)
 [ecj-lint] w = new RandomIndexWriter(random(), dir, iwc);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'w' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BasePostingsFormatTestCase.java (at line 314)
 [ecj-lint] Analyzer analyzer = new MockAnalyzer(random());
 [ecj-lint]         ^^^^^^^^
 [ecj-lint] Resource leak: 'analyzer' is never closed
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BasePostingsFormatTestCase.java (at line 539)
 [ecj-lint] LineFileDocs docs = new LineFileDocs(random());
 [ecj-lint]             ^^^^
 [ecj-lint] Resource leak: 'docs' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/index/BaseStoredFieldsFormatTestCase.java (at line 464)
 [ecj-lint] throw ex.get();
 [ecj-lint] ^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'iw' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/search/ShardSearchingTestBase.java (at line 555)
 [ecj-lint] final LineFileDocs docs = new LineFileDocs(random());
 [ecj-lint]                   ^^^^
 [ecj-lint] Resource leak: 'docs' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/store/MockDirectoryWrapper.java (at line 881)
 [ecj-lint] new IndexWriter(in, iwc).rollback();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 16. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java (at line 144)
 [ecj-lint] import static com.carrotsearch.randomizedtesting.RandomizedTest.frequently;
 [ecj-lint]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import com.carrotsearch.randomizedtesting.RandomizedTest.frequently is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 17. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1310)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1324)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1324)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1346)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1346)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1366)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/test-framework/src/java/org/apache/lucene/util/TestUtil.java (at line 1385)
 [ecj-lint] fs = ffs.getDelegate();
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fs' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 23 problems (1 error, 22 warnings)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build.xml:203: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2105: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2144: Compile failed; see the compiler error output for details.

Total time: 389 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 276 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/276/

All tests passed

Build Log:
[...truncated 466 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J0-20171208_165844_8742006765338077402132.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] #
   [junit4] # There is insufficient memory for the Java Runtime Environment to continue.
   [junit4] # Native memory allocation (mmap) failed to map 114294784 bytes for committing reserved memory.
   [junit4] # An error report file with more information is saved as:
   [junit4] # /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/J0/hs_err_pid24683.log
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J0-20171208_165844_8749039779944601907095.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000eca00000, 114294784, 0) failed; error='Cannot allocate memory' (errno=12)
   [junit4] <<< JVM J0: EOF ----

[...truncated 824 lines...]
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 142606336 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/hs_err_pid24332.log
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-7.x #274 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 277 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/277/

11 tests failed.
FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas

Error Message:
Could not load collection from ZK: TooManyReplicasInSeveralFlavors

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: TooManyReplicasInSeveralFlavors
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:60962F4041A265BE]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:110)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
        at org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas(CollectionTooManyReplicasTest.java:91)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/TooManyReplicasInSeveralFlavors/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 44 more


FAILED:  org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testRandom

Error Message:
Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason: <pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason:
<pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:9F887E97BE24EC7C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:461)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:429)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testRandom(TestCloudJSONFacetJoinDomain.java:370)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testBespoke

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection]
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:E61FDFA1C7C49DF8]:0)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:461)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:429)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testBespoke(TestCloudJSONFacetJoinDomain.java:280)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason:
<pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        ... 48 more


FAILED:  org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testMalformedGivesError

Error Message:
bogus -> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason: <pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>  expected:<400> but was:<404>

Stack Trace:
java.lang.AssertionError: bogus -> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason:
<pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
 expected:<400> but was:<404>
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:1A5BE24EBA1652FC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testMalformedGivesError(TestCloudJSONFacetJoinDomain.java:211)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestHdfsCloudBackupRestore.test

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:65906442A1B837F7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:287)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:140)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica.test

Error Message:
Indexed documents not found expected:<10> but was:<0>

Stack Trace:
java.lang.AssertionError: Indexed documents not found expected:<10> but was:<0>
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:65906442A1B837F7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica.test(TestLeaderElectionWithEmptyReplica.java:105)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testAtomicUpdateOfSegmentSortField

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:CD67C9DD901C1B7]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.TestSegmentSorting.createCollection(TestSegmentSorting.java:84)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testMetricsTag

Error Message:
KeeperErrorCode = Session expired for /autoscaling.json

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:669EBB32DC390D5F]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$setData$6(SolrZkClient.java:351)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:351)
        at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:539)
        at org.apache.solr.cloud.autoscaling.TestPolicyCloud.after(TestPolicyCloud.java:79)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:53330_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:53330_solr wasn't created
        at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:F57ED394017197E0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:930)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest

Error Message:
Could not load collection from ZK: localShardsTestColl

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: localShardsTestColl
        at __randomizedtesting.SeedInfo.seed([8FC2650D0D48DB60:730FFDB4F39F4C9A]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:429)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/localShardsTestColl/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 44 more


FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1

Error Message:
CollectionStateWatcher not notified of stateformat=1 collection creation

Stack Trace:
java.lang.AssertionError: CollectionStateWatcher not notified of stateformat=1 collection creation
        at __randomizedtesting.SeedInfo.seed([8FC2650D0D48DB60:E80579E3F9F0FFC1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1(TestCollectionStateWatchers.java:290)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11786 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionTooManyReplicasTest
   [junit4]   2> 840857 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/init-core-data-001
   [junit4]   2> 840858 WARN  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 841022 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 841023 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 841024 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001
   [junit4]   2> 841024 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 841026 INFO  (Thread-442) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 841026 INFO  (Thread-442) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 841079 ERROR (Thread-442) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 841126 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40795
   [junit4]   2> 841232 INFO  (zkConnectionManagerCallback-282-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 841268 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 843418 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 843563 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@81ed203{/solr,null,AVAILABLE}
   [junit4]   2> 843564 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3c531598{SSL,[ssl, http/1.1]}{127.0.0.1:49728}
   [junit4]   2> 843564 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.Server Started @863517ms
   [junit4]   2> 843564 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49728}
   [junit4]   2> 843565 ERROR (jetty-launcher-279-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T21:13:01.975Z
   [junit4]   2> 843571 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4b312e2e{/solr,null,AVAILABLE}
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4008ff7e{SSL,[ssl, http/1.1]}{127.0.0.1:33000}
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.Server Started @863640ms
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33000}
   [junit4]   2> 843687 ERROR (jetty-launcher-279-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 843688 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 843688 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 843688 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T21:13:02.098Z
   [junit4]   2> 843871 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@45d763ee{/solr,null,AVAILABLE}
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4c314cce{SSL,[ssl, http/1.1]}{127.0.0.1:56434}
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.Server Started @863825ms
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56434}
   [junit4]   2> 843872 ERROR (jetty-launcher-279-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T21:13:02.282Z
   [junit4]   2> 843891 INFO  (zkConnectionManagerCallback-286-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844195 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 844219 INFO  (zkConnectionManagerCallback-284-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844274 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 844322 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40795/solr
   [junit4]   2> 844414 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40795/solr
   [junit4]   2> 844414 INFO  (zkConnectionManagerCallback-288-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844451 INFO  (zkConnectionManagerCallback-296-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844634 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 844713 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40795/solr
   [junit4]   2> 844713 INFO  (zkConnectionManagerCallback-292-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844778 INFO  (zkConnectionManagerCallback-298-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844779 INFO  (zkConnectionManagerCallback-302-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 845143 INFO  (zkConnectionManagerCallback-304-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 845162 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 845166 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33000_solr
   [junit4]   2> 845182 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.Overseer Overseer (id=99140736085983240-127.0.0.1:33000_solr-n_0000000000) starting
   [junit4]   2> 845230 INFO  (zkConnectionManagerCallback-307-thread-1-processing-n:127.0.0.1:56434_solr) [n:127.0.0.1:56434_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 845290 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33000_solr
   [junit4]   2> 845354 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 845355 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56434_solr
   [junit4]   2> 845402 INFO  (zkCallback-303-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 845557 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 845559 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 845560 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49728_solr
   [junit4]   2> 846199 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33000.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 846213 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33000.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 846213 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33000.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 846222 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node3/.
   [junit4]   2> 846922 INFO  (zkCallback-303-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 847269 INFO  (zkCallback-306-thread-1-processing-n:127.0.0.1:56434_solr) [n:127.0.0.1:56434_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 847451 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 847872 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56434.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848114 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56434.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848114 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56434.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848115 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node1/.
   [junit4]   2> 848541 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848562 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 848672 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848673 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node2/.
   [junit4]   2> 849302 INFO  (zkConnectionManagerCallback-312-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 849422 INFO  (zkConnectionManagerCallback-316-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 849423 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 849424 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40795/solr ready
   [junit4]   2> 849651 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDownedShards
   [junit4]   2> 849670 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 850145 INFO  (qtp1741440036-1502) [n:127.0.0.1:56434_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=shardstart&replicationFactor=1&collection.configName=conf&maxShardsPerNode=2&name=TooManyReplicasWhenAddingDownedNode&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 850214 INFO  (OverseerThreadFactory-451-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.CreateCollectionCmd Create collection TooManyReplicasWhenAddingDownedNode
   [junit4]   2> 850363 INFO  (OverseerStateUpdate-99140736085983240-127.0.0.1:33000_solr-n_0000000000) [n:127.0.0.1:33000_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"TooManyReplicasWhenAddingDownedNode",
   [junit4]   2>   "shard":"shardstart",
   [junit4]   2>   "core":"TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:49728/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 851140 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1&action=CREATE&numShards=1&collection=TooManyReplicasWhenAddingDownedNode&shard=shardstart&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 851140 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 853894 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 854222 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 855275 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 855373 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.IndexSchema [TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] Schema name=minimal
   [junit4]   2> 855375 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 855375 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1' using configuration from collection TooManyReplicasWhenAddingDownedNode, trusted=true
   [junit4]   2> 855376 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.core.TooManyReplicasWhenAddingDownedNode.shardstart.replica_n1' (registry 'solr.core.TooManyReplicasWhenAddingDownedNode.shardstart.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 855376 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 855376 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore [[TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node2/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node2/./TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/data/]
   [junit4]   2> 857419 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 857419 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 857420 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857420 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857466 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5dadb795[TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] main]
   [junit4]   2> 857475 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 857475 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 857476 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 857476 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586251794623758336
   [junit4]   2> 857577 INFO  (searcherExecutor-458-thread-1-processing-n:127.0.0.1:49728_solr x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1 s:shardstart c:TooManyReplicasWhenAddingDownedNode r:core_node2) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore [TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] Registered new searcher Searcher@5dadb795[TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:49728/solr/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:49728/solr/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/ has no replicas
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 857585 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:49728/solr/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/ shardstart
   [junit4]   2> 857944 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 857944 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 857954 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 857970 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1&action=CREATE&numShards=1&collection=TooManyReplicasWhenAddingDownedNode&shard=shardstart&wt=javabin&version=2&replicaType=NRT} status=0 QTime=6830
   [junit4]   2> 858091 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 858091 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 860211 INFO  (qtp1741440036-1502) [n:127.0.0.1:56434_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 860219 INFO  (OverseerCollectionConfigSetProcessor-99140736085983240-127.0.0.1:33000_solr-n_0000000000) [n:127.0.0.1:33000_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 860231 INFO  (qtp1741440036-1502) [n:127.0.0.1:56434_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=shardstart&replicationFactor=1&collection.configName=conf&maxShardsPerNode=2&name=TooManyReplicasWhenAddingDownedNode&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2} status=0 QTime=10085
   [junit4]   2> 860334 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3c531598{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 860338 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1340382632
   [junit4]   2> 860338 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 860339 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@61de3b4e: rootName = solr_49728, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2ddf3186
   [junit4]   2> 860450 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 860451 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3c26073e: rootName = solr_49728, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2de3c241
   [junit4]   2> 860621 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 860621 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@19dd1edb: rootName = solr_49728, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5ca1b7f2
   [junit4]   2> 860741 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 860750 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 860750 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:49728_solr
   [junit4]   2> 860752 INFO  (zkCallback-306-thread-1-processing-n:127.0.0.1:56434_solr) [n:127.0.0.1:56434_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860752 INFO  (zkCallback-315-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860752 INFO  (zkCallback-315-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860761 INFO  (zkCallback-303-thread-2-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860762 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860763 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 860986 INFO  (coreCloseExecutor-463-thread-1) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore [TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@76cd70c2
   [junit4]   2> 860986 INFO  (coreCloseExecutor-463-thread-1) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.TooManyReplicasWhenAddingDownedNode.shardstart.replica_n1, tag=1993175234
   [junit4]   2> 860986 INFO  (coreCloseExecutor-463-thread-1) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_sh

[...truncated too long message...]

l, agent id = null] for registry solr.core.localShardsTestColl.shard2.replica_n8 / com.codahale.metrics.MetricRegistry@7ba5f5c0
   [junit4]   2> 465346 INFO  (coreCloseExecutor-357-thread-6) [n:127.0.0.1:45012_solr c:foo s:shard1 r:core_node6 x:foo_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.foo.shard1.leader, tag=1094082352
   [junit4]   2> 465346 INFO  (coreCloseExecutor-357-thread-5) [n:127.0.0.1:45012_solr c:multicollection1 s:shard1 r:core_node3 x:multicollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection1.shard1.leader, tag=53893817
   [junit4]   2> 465346 INFO  (coreCloseExecutor-357-thread-4) [n:127.0.0.1:45012_solr c:localShardsTestColl s:shard3 r:core_node18 x:localShardsTestColl_shard3_replica_n16] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=911787225
   [junit4]   2> 465366 INFO  (coreCloseExecutor-357-thread-3) [n:127.0.0.1:45012_solr c:localShardsTestColl s:shard1 r:core_node6 x:localShardsTestColl_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=563712393
   [junit4]   2> 465367 INFO  (coreCloseExecutor-355-thread-4) [n:127.0.0.1:53893_solr c:multicollection2 s:shard2 r:core_node4 x:multicollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection2.shard2.leader, tag=226728099
   [junit4]   2> 465369 INFO  (coreCloseExecutor-355-thread-1) [n:127.0.0.1:53893_solr c:localShardsTestColl s:shard2 r:core_node11 x:localShardsTestColl_shard2_replica_n8] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard2.leader, tag=1241546616
   [junit4]   2> 465369 INFO  (coreCloseExecutor-355-thread-2) [n:127.0.0.1:53893_solr c:localShardsTestColl s:shard3 r:core_node17 x:localShardsTestColl_shard3_replica_n14] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=531793602
   [junit4]   2> 465369 INFO  (coreCloseExecutor-355-thread-3) [n:127.0.0.1:53893_solr c:localShardsTestColl s:shard1 r:core_node4 x:localShardsTestColl_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=701305072
   [junit4]   2> 465383 INFO  (coreCloseExecutor-356-thread-8) [n:127.0.0.1:51502_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.overwrite.shard1.replica_n1, tag=728896613
   [junit4]   2> 465383 INFO  (coreCloseExecutor-356-thread-8) [n:127.0.0.1:51502_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@727feb3b: rootName = solr_51502, domain = solr.core.overwrite.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.overwrite.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@5e5fa932
   [junit4]   2> 465411 INFO  (coreCloseExecutor-356-thread-8) [n:127.0.0.1:51502_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.overwrite.shard1.leader, tag=728896613
   [junit4]   2> 465417 INFO  (coreCloseExecutor-356-thread-2) [n:127.0.0.1:51502_solr c:localShardsTestColl s:shard3 r:core_node15 x:localShardsTestColl_shard3_replica_n12] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=1759574454
   [junit4]   2> 465418 INFO  (coreCloseExecutor-356-thread-4) [n:127.0.0.1:51502_solr c:localShardsTestColl s:shard1 r:core_node2 x:localShardsTestColl_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=90092253
   [junit4]   2> 465418 INFO  (coreCloseExecutor-356-thread-3) [n:127.0.0.1:51502_solr c:localShardsTestColl s:shard2 r:core_node9 x:localShardsTestColl_shard2_replica_n7] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard2.leader, tag=927970669
   [junit4]   2> 465418 INFO  (coreCloseExecutor-356-thread-5) [n:127.0.0.1:51502_solr c:multicollection1 s:shard2 r:core_node4 x:multicollection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection1.shard2.leader, tag=1375829709
   [junit4]   2> 465425 INFO  (coreCloseExecutor-356-thread-7) [n:127.0.0.1:51502_solr c:foo s:shard1 r:core_node4 x:foo_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.foo.shard1.leader, tag=1372373832
   [junit4]   2> 465950 INFO  (jetty-closer-149-thread-2) [    ] o.a.s.c.Overseer Overseer (id=99141425872896006-127.0.0.1:53893_solr-n_0000000000) closing
   [junit4]   2> 465951 INFO  (OverseerStateUpdate-99141425872896006-127.0.0.1:53893_solr-n_0000000000) [n:127.0.0.1:53893_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:53893_solr
   [junit4]   2> 466068 INFO  (jetty-closer-149-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@20def09d{/solr,null,UNAVAILABLE}
   [junit4]   2> 466069 WARN  (OverseerAutoScalingTriggerThread-99141425872896006-127.0.0.1:53893_solr-n_0000000000) [n:127.0.0.1:53893_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 466070 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1603899ffae0006, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 466329 INFO  (jetty-closer-149-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5fcfe4d0{/solr,null,UNAVAILABLE}
   [junit4]   2> 467849 INFO  (jetty-closer-149-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@48a5fe5{/solr,null,UNAVAILABLE}
   [junit4]   2> 467878 ERROR (SUITE-CloudSolrClientTest-seed#[8FC2650D0D48DB60]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 467878 INFO  (SUITE-CloudSolrClientTest-seed#[8FC2650D0D48DB60]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38623 38623
   [junit4]   2> 467990 INFO  (Thread-115) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38623 38623
   [junit4]   2> 468013 WARN  (Thread-115) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 9 /solr/clusterprops.json
   [junit4]   2> 9 /solr/aliases.json
   [junit4]   2> 3 /solr/security.json
   [junit4]   2> 3 /solr/configs/conf
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 42 /solr/collections/localShardsTestColl/state.json
   [junit4]   2> 24 /solr/collections/foo/state.json
   [junit4]   2> 18 /solr/collections/stale_state_test_col/state.json
   [junit4]   2> 18 /solr/collections/multicollection2/state.json
   [junit4]   2> 16 /solr/collections/collection1/state.json
   [junit4]   2> 16 /solr/collections/2nd_collection/state.json
   [junit4]   2> 14 /solr/collections/nemesis/state.json
   [junit4]   2> 13 /solr/collections/multicollection1/state.json
   [junit4]   2> 9 /solr/clusterstate.json
   [junit4]   2> 6 /solr/collections/overwrite/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 9 /solr/live_nodes
   [junit4]   2> 9 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_8FC2650D0D48DB60-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), title_s=Lucene50(blocksize=128), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), text=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting), multiDefault=DocValuesFormat(name=Lucene70), title_s=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=344, maxMBSortInHeap=5.990646958084055, sim=RandomSimilarity(queryNorm=false): {}, locale=en-IE, timezone=America/Argentina/San_Juan
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=174697888,total=396886016
   [junit4]   2> NOTE: All tests run in this JVM: [HyperbolicSineEvaluatorTest, JdbcDriverTest, LBHttpSolrClientBuilderTest, SineEvaluatorTest, DocumentAnalysisResponseTest, ExclusiveOrEvaluatorTest, NormalizeEvaluatorTest, UuidEvaluatorTest, LessThanEqualToEvaluatorTest, DivideEvaluatorTest, PowerEvaluatorTest, TestCollectionStateWatchers, SolrExceptionTest, TestSuggesterResponse, CoalesceEvaluatorTest, NoOpResponseParserTest, SolrExampleBinaryTest, ArcCosineEvaluatorTest, TestCloudSolrClientConnections, CloudSolrClientTest]
   [junit4] Completed [73/146 (2!)] on J2 in 162.71s, 16 tests, 1 error <<< FAILURES!

[...truncated 46663 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 278 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/278/

3 tests failed.
FAILED:  org.apache.solr.cloud.DistributedQueueTest.testPeekElements

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([91D5E80E9902C4B:F433E4A139A97856]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.DistributedQueueTest.testPeekElements(DistributedQueueTest.java:265)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.test

Error Message:
Error from server at https://127.0.0.1:33985/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000005

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:33985/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000005
        at __randomizedtesting.SeedInfo.seed([91D5E80E9902C4B:8149615A476C41B3]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:444)
        at org.apache.solr.cloud.MoveReplicaTest.beforeTest(MoveReplicaTest.java:95)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelCommitStream

Error Message:
expected:<5> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<2>
        at __randomizedtesting.SeedInfo.seed([72515E0A109A540E:52BB3C0A8CDBB942]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelCommitStream(StreamExpressionTest.java:4594)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11771 lines...]
   [junit4] Suite: org.apache.solr.cloud.DistributedQueueTest
   [junit4]   2> 663149 INFO  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.DistributedQueueTest_91D5E80E9902C4B-001/init-core-data-001
   [junit4]   2> 663150 WARN  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2171 numCloses=2171
   [junit4]   2> 663150 INFO  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 663151 INFO  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 663212 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLeakChildWatcher
   [junit4]   2> 663212 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 663212 INFO  (Thread-84) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 663212 INFO  (Thread-84) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 663466 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:40620
   [junit4]   2> 663483 ERROR (Thread-84) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 663637 INFO  (zkConnectionManagerCallback-85-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 663925 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLeakChildWatcher
   [junit4]   2> 663932 ERROR (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 663932 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40620 40620
   [junit4]   2> 664203 INFO  (Thread-84) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40620 40620
   [junit4]   2> 664324 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPeekElements
   [junit4]   2> 664324 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 664362 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 664362 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 664439 ERROR (Thread-85) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 664470 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:55309
   [junit4]   2> 664606 INFO  (zkConnectionManagerCallback-88-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 666729 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPeekElements
   [junit4]   2> 666733 ERROR (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 666733 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55309 55309
   [junit4]   2> 666919 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55309 55309
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedQueueTest -Dtests.method=testPeekElements -Dtests.seed=91D5E80E9902C4B -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-BH -Dtests.timezone=NST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 2.72s J0 | DistributedQueueTest.testPeekElements <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([91D5E80E9902C4B:F433E4A139A97856]:0)
   [junit4]    > at org.apache.solr.cloud.DistributedQueueTest.testPeekElements(DistributedQueueTest.java:265)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 666965 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueue
   [junit4]   2> 666966 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 666966 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 666966 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 667626 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:51431
   [junit4]   2> 667766 ERROR (Thread-86) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 667855 INFO  (zkConnectionManagerCallback-91-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 670101 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue
   [junit4]   2> 670139 ERROR (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 670139 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51431 51431
   [junit4]   2> 670475 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51431 51431
   [junit4]   2> 670477 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueueCache
   [junit4]   2> 670477 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 670482 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 670482 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 670586 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:42368
   [junit4]   2> 671036 ERROR (Thread-89) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 672158 INFO  (zkConnectionManagerCallback-94-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 672190 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueueCache
   [junit4]   2> 672202 ERROR (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 672203 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42368 42368
   [junit4]   2> 672439 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42368 42368
   [junit4]   2> 672454 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueueBlocking
   [junit4]   2> 672455 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 672498 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 672499 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 672500 ERROR (Thread-90) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 672618 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:54368
   [junit4]   2> 672678 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 674118 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46e15243 name: ZooKeeperConnection Watcher:127.0.0.1:54368/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 674118 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 676159 WARN  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]-SendThread(127.0.0.1:54368)) [    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x1603c0856050000 has expired
   [junit4]   2> 676159 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46e15243 name: ZooKeeperConnection Watcher:127.0.0.1:54368/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 676159 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 676166 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1-EventThread) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 677438 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueueBlocking
   [junit4]   2> 677447 ERROR (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 677447 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54368 54368
   [junit4]   2> 678114 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54368 54368
   [junit4]   2> 678115 WARN  (Thread-90) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/distqueue/test
   [junit4]   2>
   [junit4]   2> 678227 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLocallyOffer
   [junit4]   2> 678228 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 678273 INFO  (Thread-91) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 678273 INFO  (Thread-91) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 678326 ERROR (Thread-91) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 678412 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:37747
   [junit4]   2> 678978 INFO  (zkConnectionManagerCallback-100-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 681873 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLocallyOffer
   [junit4]   2> 681891 ERROR (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 681891 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37747 37747
   [junit4]   2> 682275 INFO  (Thread-91) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37747 37747
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.DistributedQueueTest_91D5E80E9902C4B-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-BH, timezone=NST
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=94191480,total=305659904
   [junit4]   2> NOTE: All tests run in this JVM: [UpdateParamsTest, TestFilteredDocIdSet, QueryParsingTest, RulesTest, TestJsonFacets, DistributedQueueTest]
   [junit4] Completed [32/751 (1!)] on J0 in 22.08s, 6 tests, 1 failure <<< FAILURES!

[...truncated 1792 lines...]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4]   2> 7515951 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/init-core-data-001
   [junit4]   2> 7515951 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 7515952 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 7515953 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 7515954 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001
   [junit4]   2> 7515954 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 7515954 INFO  (Thread-3615) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 7515954 INFO  (Thread-3615) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 7515955 ERROR (Thread-3615) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 7516057 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:45661
   [junit4]   2> 7516063 INFO  (zkConnectionManagerCallback-3632-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516093 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516094 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516098 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516103 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41c5a387{/solr,null,AVAILABLE}
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@20c6fff2{SSL,[ssl, http/1.1]}{127.0.0.1:36881}
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.Server Started @7533075ms
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36881}
   [junit4]   2> 7516112 ERROR (jetty-launcher-3629-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.822Z
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@76baca41{/solr,null,AVAILABLE}
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@16ae16c1{SSL,[ssl, http/1.1]}{127.0.0.1:52835}
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.Server Started @7533078ms
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52835}
   [junit4]   2> 7516114 ERROR (jetty-launcher-3629-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.824Z
   [junit4]   2> 7516134 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@cff8903{/solr,null,AVAILABLE}
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c771c69{SSL,[ssl, http/1.1]}{127.0.0.1:48847}
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.Server Started @7533099ms
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48847}
   [junit4]   2> 7516135 ERROR (jetty-launcher-3629-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.845Z
   [junit4]   2> 7516215 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7eb59e2{/solr,null,AVAILABLE}
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74e9fb0e{SSL,[ssl, http/1.1]}{127.0.0.1:55369}
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.Server Started @7533180ms
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55369}
   [junit4]   2> 7516216 ERROR (jetty-launcher-3629-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.926Z
   [junit4]   2> 7516247 INFO  (zkConnectionManagerCallback-3634-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516247 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516316 INFO  (zkConnectionManagerCallback-3638-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516327 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516339 INFO  (zkConnectionManagerCallback-3636-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516340 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516454 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516455 INFO  (zkConnectionManagerCallback-3640-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516482 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516482 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516483 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516590 INFO  (zkConnectionManagerCallback-3652-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516577 INFO  (zkConnectionManagerCallback-3644-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516577 INFO  (zkConnectionManagerCallback-3650-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516738 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516758 INFO  (zkConnectionManagerCallback-3656-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516806 INFO  (zkConnectionManagerCallback-3660-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516810 INFO  (zkConnectionManagerCallback-3658-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516811 INFO  (zkConnectionManagerCallback-3662-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516906 INFO  (zkConnectionManagerCallback-3666-thread-1-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516992 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7516992 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55369_solr
   [junit4]   2> 7516994 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.Overseer Overseer (id=99145646713405452-127.0.0.1:55369_solr-n_0000000000) starting
   [junit4]   2> 7516995 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7516996 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36881_solr
   [junit4]   2> 7517018 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 7517055 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55369_solr
   [junit4]   2> 7517055 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 7517114 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 7517127 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 7517246 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 7517249 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7517311 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 7517312 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7517331 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48847_solr
   [junit4]   2> 7517334 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517380 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517387 INFO  (zkCallback-3657-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517387 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517398 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52835_solr
   [junit4]   2> 7517510 INFO  (zkCallback-3657-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7517510 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7517510 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7517598 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7519731 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519783 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519783 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519784 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node3/.
   [junit4]   2> 7519896 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519964 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519964 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519965 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node4/.
   [junit4]   2> 7520008 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520010 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48847.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520319 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520320 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520320 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node2/.
   [junit4]   2> 7520324 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48847.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520324 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48847.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520325 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node1/.
   [junit4]   2> 7520731 INFO  (zkConnectionManagerCallback-3671-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7520734 INFO  (zkConnectionManagerCallback-3675-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7520734 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 7520735 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45661/solr ready
   [junit4]   2> 7520907 INFO  (qtp1026759654-17396) [n:127.0.0.1:55369_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 7521014 INFO  (qtp1026759654-17396) [n:127.0.0.1:55369_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=107
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 7521961 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 7522102 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7522104 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7522235 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_43029_hdfs____.bh5gam/webapp
   [junit4]   2> 7523053 INFO  (OverseerCollectionConfigSetProcessor-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 7524743 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43029
   [junit4]   2> 7529954 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7529955 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7530342 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46013_datanode____alc31z/webapp
   [junit4]   2> 7533637 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46013
   [junit4]   2> 7534968 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7535019 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7535180 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36421_datanode____i93l7b/webapp
   [junit4]   2> 7536491 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:40621) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 7536714 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e85555a10eb1a: from storage DS-4c393cb1-2143-4f8b-8754-cc7e80df835e node DatanodeRegistration(127.0.0.1:50668, datanodeUuid=c91443d4-2152-4d5c-a5a2-1239693cb7a8, infoPort=57431, infoSecurePort=0, ipcPort=34522, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 7536714 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e85555a10eb1a: from storage DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a node DatanodeRegistration(127.0.0.1:50668, datanodeUuid=c91443d4-2152-4d5c-a5a2-1239693cb7a8, infoPort=57431, infoSecurePort=0, ipcPort=34522, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 7537916 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36421
   [junit4]   2> 7539143 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:40621) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 7539301 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e8555ec8dc4a4: from storage DS-08a50d7e-6aa0-475a-bfb5-c65329252d52 node DatanodeRegistration(127.0.0.1:41673, datanodeUuid=0f9bea97-fc0c-4fd5-82b0-29b83f87a9bc, infoPort=56167, infoSecurePort=0, ipcPort=37351, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 7539302 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e8555ec8dc4a4: from storage DS-3996a272-06ee-47ec-804c-a1e76c369319 node DatanodeRegistration(127.0.0.1:41673, datanodeUuid=0f9bea97-fc0c-4fd5-82b0-29b83f87a9bc, infoPort=56167, infoSecurePort=0, ipcPort=37351, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 7540799 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFailedMove
   [junit4]   2> 7540800 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 7540801 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7540946 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e35d4cf{/solr,null,AVAILABLE}
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@71f127a7{SSL,[ssl, http/1.1]}{127.0.0.1:33985}
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.Server Started @7557911ms
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33985}
   [junit4]   2> 7540947 ERROR (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 7540948 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7540948 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7540948 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:02:14.658Z
   [junit4]   2> 7540974 INFO  (zkConnectionManagerCallback-3678-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7540975 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7541032 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7541051 INFO  (zkConnectionManagerCallback-3682-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7541155 INFO  (zkConnectionManagerCallback-3684-thread-1-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7541182 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 7541203 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7541218 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33985_solr
   [junit4]   2> 7541220 INFO  (zkCallback-3674-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541235 INFO  (zkCallback-3657-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541235 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541235 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541279 INFO  (zkCallback-3674-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541279 INFO  (zkCallback-3665-thread-1-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541294 INFO  (zkCallback-3683-thread-1-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7542349 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7542391 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7542391 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7542393 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node5/.
   [junit4]   2> 7543358 INFO  (zkConnectionManagerCallback-3687-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7543613 INFO  (qtp701827325-17637) [n:127.0.0.1:33985_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&autoAddReplicas=false&name=MoveReplicaHDFSTest_failed_coll_true&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 7543646 INFO  (OverseerThreadFactory-7718-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.CreateCollectionCmd Create collection MoveReplicaHDFSTest_failed_coll_true
   [junit4]   2> 7543904 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33985/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 7543971 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:55369/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 7543983 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:52835/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 7543998 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36881/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 7544518 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544518 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544519 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544520 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544563 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544563 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544563 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544563 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544679 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544679 INFO  (zkCallback-3665-thread-1-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544679 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544682 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544682 INFO  (zkCallback-3683-thread-1-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544683 INFO  (zkCallback-3661-thread-2-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544683 INFO  (zkCallback-3683-thread-2-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544684 INFO  (zkCallback-3659-thread-2-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7545599 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7545615 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7545900 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7545904 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] Schema name=minimal
   [junit4]   2> 7545906 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7545906 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7545907 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n3' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7545936 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 7548091 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7548092 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7548092 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n1' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7548100 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] Schema name=minimal
   [junit4]   2> 7548102 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7548102 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7548111 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7548112 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7548112 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7548112 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node4/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data/]
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n7' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n7') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node3/MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/]
   [junit4]   2> 7548161 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/snapshot_metadata
   [junit4]   2> 7548114 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7548178 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7548178 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7548178 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node5/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data/]
   [junit4]   2> 7548179 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data/snapshot_metadata
   [junit4]   2> 7548308 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data/snapshot_metadata
   [junit4]   2> 7548315 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7549318 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] Schema name=minimal
   [junit4]   2> 7549321 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7549321 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7549321 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n5' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node2/MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data/]
   [junit4]   2> 7549448 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data
   [junit4]   2> 7549467 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data/snapshot_metadata
   [junit4]   2> 7549638 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data
   [junit4]   2> 7549654 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data
   [junit4]   2> 7549828 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data
   [junit4]   2> 7549968 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data/index
   [junit4]   2> 7550034 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/index
   [junit4]   2> 7550055 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data/index
   [junit4]   2> 7550402 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data/index
   [junit4]   2> 7552937 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741827_1003{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c393cb1-2143-4f8b-8754-cc7e80df835e:NORMAL:127.0.0.1:50668|RBW], ReplicaUC[[DISK]DS-3996a272-06ee-47ec-804c-a1e76c369319:NORMAL:127.0.0.1:41673|RBW]]} size 69
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-08a50d7e-6aa0-475a-bfb5-c65329252d52:NORMAL:127.0.0.1:41673|RBW], ReplicaUC[[DISK]DS-4c393cb1-2143-4f8b-8754-cc7e80df835e:NORMAL:127.0.0.1:50668|RBW]]} size 0
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-08a50d7e-6aa0-475a-bfb5-c65329252d52:NORMAL:127.0.0.1:41673|RBW], ReplicaUC[[DISK]DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a:NORMAL:127.0.0.1:50668|RBW]]} size 69
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3996a272-06ee-47ec-804c-a1e76c369319:NORMAL:127.0.0.1:41673|RBW], ReplicaUC[[DISK]DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a:NORMAL:127.0.0.1:50668|RBW]]} size 0
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741827_1003 size 69
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741825_1001 size 69
   [junit4]   2> 7553154 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741828_1004 size 69
   [junit4]   2> 7553155 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a:NORMAL:127.0.0.1:50668|RBW], ReplicaUC[[DISK]DS-08a50d7e-6aa0-475a-bfb5-c65329252d52:NORMAL:127.0.0.1:41673|RBW]]} size 0
   [junit4]   2> 7554056 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 7554056 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 7554056 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 7554157 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 7554157 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 7554157 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:

[...truncated too long message...]

ted live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 1020650 INFO  (zkCallback-272-thread-5-processing-n:127.0.0.1:36238_solr) [n:127.0.0.1:36238_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 1020661 INFO  (jetty-closer-236-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1020661 INFO  (jetty-closer-236-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1020661 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:37583_solr x:collection1_collection_shard2_replica_n2 s:shard2 c:collection1_collection r:core_node4) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 1020665 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@50e5f18b
   [junit4]   2> 1020665 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1_collection.shard1.replica_n1, tag=1357246859
   [junit4]   2> 1020692 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51bdd4e8: rootName = solr_36238, domain = solr.core.collection1_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection1_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@7063501e
   [junit4]   2> 1020707 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@3f61282a
   [junit4]   2> 1020710 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard1.leader, tag=1357246859
   [junit4]   2> 1020732 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1_collection.shard2.replica_n2, tag=1063331882
   [junit4]   2> 1020732 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@41dfc7bb: rootName = solr_37583, domain = solr.core.collection1_collection.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.collection1_collection.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@10d0f746
   [junit4]   2> 1020755 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard2.leader, tag=1063331882
   [junit4]   2> 1020784 INFO  (jetty-closer-236-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2a8bc0d6{/solr,null,UNAVAILABLE}
   [junit4]   2> 1020794 INFO  (jetty-closer-236-thread-1) [    ] o.a.s.c.Overseer Overseer (id=99145935423864841-127.0.0.1:37583_solr-n_0000000000) closing
   [junit4]   2> 1020794 INFO  (OverseerStateUpdate-99145935423864841-127.0.0.1:37583_solr-n_0000000000) [n:127.0.0.1:37583_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37583_solr
   [junit4]   2> 1020816 WARN  (OverseerAutoScalingTriggerThread-99145935423864841-127.0.0.1:37583_solr-n_0000000000) [n:127.0.0.1:37583_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1020844 INFO  (jetty-closer-236-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@12d318bc{/solr,null,UNAVAILABLE}
   [junit4]   2> 1020857 ERROR (SUITE-StreamExpressionTest-seed#[72515E0A109A540E]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1020858 INFO  (SUITE-StreamExpressionTest-seed#[72515E0A109A540E]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51520 51520
   [junit4]   2> 1021031 INFO  (Thread-196) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51520 51520
   [junit4]   2> 1021032 WARN  (Thread-196) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 79 /solr/aliases.json
   [junit4]   2> 11 /solr/configs/conf
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 79 /solr/clusterprops.json
   [junit4]   2> 79 /solr/clusterstate.json
   [junit4]   2> 26 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 22 /solr/collections/destinationCollection/state.json
   [junit4]   2> 20 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 17 /solr/collections/collection1_collection/state.json
   [junit4]   2> 14 /solr/collections/mainCorpus1/state.json
   [junit4]   2> 14 /solr/collections/destination1/state.json
   [junit4]   2> 14 /solr/collections/workQueue/state.json
   [junit4]   2> 12 /solr/collections/mainCorpus/state.json
   [junit4]   2> 10 /solr/collections/destination/state.json
   [junit4]   2> 10 /solr/collections/workQueue1/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/99145935423864841-127.0.0.1:37583_solr-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 79 /solr/live_nodes
   [junit4]   2> 79 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_72515E0A109A540E-001
   [junit4]   2> Dec 09, 2017 7:21:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=FSTOrd50, join2_s=BlockTreeOrds(blocksize=128), multiDefault=BlockTreeOrds(blocksize=128), subject=BlockTreeOrds(blocksize=128), a1_s=Lucene50(blocksize=128), a2_s=FSTOrd50, test_t=Lucene50(blocksize=128), level1_s=FSTOrd50, ident_s=Lucene50(blocksize=128), body_t=Lucene50(blocksize=128), s_multi=FSTOrd50, a_ss=BlockTreeOrds(blocksize=128), col_s=Lucene50(blocksize=128), level2_s=Lucene50(blocksize=128), a_s=FSTOrd50, a_t=BlockTreeOrds(blocksize=128), side_s=Lucene50(blocksize=128), id=Lucene50(blocksize=128), text=FSTOrd50}, docValues:{expr_s=DocValuesFormat(name=Lucene70), join2_s=DocValuesFormat(name=Asserting), multiDefault=DocValuesFormat(name=Asserting), a1_s=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Asserting), a_f=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Lucene70), join1_i=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Memory), join3_i=DocValuesFormat(name=Memory), test_i=DocValuesFormat(name=Asserting), field_i=DocValuesFormat(name=Asserting), level1_s=DocValuesFormat(name=Lucene70), s_multi=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Asserting), col_s=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Lucene70), a_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Asserting), i_multi=DocValuesFormat(name=Memory), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Lucene70), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Memory), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1578, maxMBSortInHeap=6.435731837867215, sim=RandomSimilarity(queryNorm=false): {}, locale=lt-LT, timezone=Etc/GMT+11
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=173714128,total=423624704
   [junit4]   2> NOTE: All tests run in this JVM: [TestSuggesterResponse, NormalizeEvaluatorTest, SelectWithEvaluatorsTest, PowerEvaluatorTest, HttpSolrClientBuilderTest, SolrExceptionTest, TestJsonRecordReader, TestCollectionAdminRequest, RegressionEvaluatorTest, MultiplyEvaluatorTest, JettyWebappTest, UuidEvaluatorTest, CloudSolrClientBuilderTest, NoOpResponseParserTest, TestCollectionStateWatchers, SolrParamTest, CommonParamsTest, SolrExampleXMLTest, SolrExampleBinaryTest, AddEvaluatorTest, LBHttpSolrClientBuilderTest, GreaterThanEqualToEvaluatorTest, LessThanEqualToEvaluatorTest, SolrExampleJettyTest, CoalesceEvaluatorTest, TestPolicy, CloudSolrClientCacheTest, CloudSolrClientMultiConstructorTest, CloudSolrClientTest, StreamExpressionTest]
   [junit4] Completed [146/146 (1!)] on J2 in 360.24s, 139 tests, 1 failure, 9 skipped <<< FAILURES!

[...truncated 46421 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 279 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/279/

7 tests failed.
FAILED:  org.apache.lucene.classification.BooleanPerceptronClassifierTest.testPerformance

Error Message:
training took more than 10s: 10s

Stack Trace:
java.lang.AssertionError: training took more than 10s: 10s
        at __randomizedtesting.SeedInfo.seed([EA7BEF4AA32652D4:2D9A1D68C8926A7B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.lucene.classification.BooleanPerceptronClassifierTest.testPerformance(BooleanPerceptronClassifierTest.java:93)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:A493FE78D98F17A0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:279)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudSearcherWarming.testRepFactor1LeaderStartup

Error Message:
Error from server at http://127.0.0.1:36410/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000012

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36410/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000012
        at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:8AF6200FF41A13C4]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.TestCloudSearcherWarming.testRepFactor1LeaderStartup(TestCloudSearcherWarming.java:100)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testStopAllStartAll

Error Message:
Could not load collection from ZK: testStopAllStartAllCollection

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: testStopAllStartAllCollection
        at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:2BE072A4A4CDF705]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:365)
        at org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testStopAllStartAll(TestCollectionsAPIViaSolrCloudCluster.java:231)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testStopAllStartAllCollection/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 42 more


FAILED:  org.apache.solr.cloud.TestCryptoKeys.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":30314},   "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",     "code":500}}

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":30314},
  "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}

        at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:D58A520D4B0637D2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:185)
        at org.apache.solr.cloud.TestCryptoKeys.test(TestCryptoKeys.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorCloud

Error Message:
Error from server at https://127.0.0.1:34960/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34960/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
        at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorCloud.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test

Error Message:
wrong number of events added to .system expected:<9> but was:<10>

Stack Trace:
java.lang.AssertionError: wrong number of events added to .system expected:<9> but was:<10>
        at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:D58A520D4B0637D2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test(SystemLogListenerTest.java:166)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 6151 lines...]
   [junit4] Suite: org.apache.lucene.classification.BooleanPerceptronClassifierTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BooleanPerceptronClassifierTest -Dtests.method=testPerformance -Dtests.seed=EA7BEF4AA32652D4 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=es-CO -Dtests.timezone=Pacific/Yap -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 15.5s J1 | BooleanPerceptronClassifierTest.testPerformance <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: training took more than 10s: 10s
   [junit4]    > at __randomizedtesting.SeedInfo.seed([EA7BEF4AA32652D4:2D9A1D68C8926A7B]:0)
   [junit4]    > at org.apache.lucene.classification.BooleanPerceptronClassifierTest.testPerformance(BooleanPerceptronClassifierTest.java:93)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true): {bool=IB SPL-D1, cat=ClassicSimilarity, text=LM Jelinek-Mercer(0.700000)}, locale=es-CO, timezone=Pacific/Yap
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=206667792,total=253231104
   [junit4]   2> NOTE: All tests run in this JVM: [DocToDoubleVectorUtilsTest, BooleanPerceptronClassifierTest]
   [junit4] Completed [3/11 (1!)] on J1 in 18.08s, 4 tests, 1 failure <<< FAILURES!

[...truncated 5920 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestCryptoKeys
   [junit4]   2> 2222018 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/init-core-data-001
   [junit4]   2> 2222019 WARN  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=11 numCloses=11
   [junit4]   2> 2222019 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2222078 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2222080 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2222238 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2222633 INFO  (Thread-955) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2222633 INFO  (Thread-955) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2222646 ERROR (Thread-955) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2222733 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkTestServer start zk server on port:43531
   [junit4]   2> 2222734 INFO  (zkConnectionManagerCallback-653-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223301 INFO  (zkConnectionManagerCallback-655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223313 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2223321 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2223329 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2223333 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2223337 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2223341 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2223369 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2223373 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2223385 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2223393 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2223401 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2223522 INFO  (zkConnectionManagerCallback-658-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223533 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1603f0fb0d10002, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2223541 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2225244 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2225253 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7e923269{/,null,AVAILABLE}
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3041241b{SSL,[ssl, http/1.1]}{127.0.0.1:37799}
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server Started @2243105ms
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/tempDir-001/control/data, hostContext=/, hostPort=37799, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores}
   [junit4]   2> 2225255 ERROR (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2225256 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2225256 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-10T06:14:43.727Z
   [junit4]   2> 2225457 INFO  (zkConnectionManagerCallback-660-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2225488 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2225488 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/solr.xml
   [junit4]   2> 2225492 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2225492 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2225493 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2225553 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43531/solr
   [junit4]   2> 2225661 INFO  (zkConnectionManagerCallback-664-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2225841 INFO  (zkConnectionManagerCallback-666-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226281 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2226289 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37799_
   [junit4]   2> 2226290 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.Overseer Overseer (id=99148528425697285-127.0.0.1:37799_-n_0000000000) starting
   [junit4]   2> 2227349 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37799_
   [junit4]   2> 2227440 INFO  (zkCallback-665-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2228432 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2228544 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2228545 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2228696 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores
   [junit4]   2> 2228822 INFO  (zkConnectionManagerCallback-672-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2228823 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2228824 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43531/solr ready
   [junit4]   2> 2230711 INFO  (qtp1677638809-3865) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37799_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2230785 INFO  (OverseerThreadFactory-2045-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2231206 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2231207 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2231361 INFO  (zkCallback-665-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2234704 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2234761 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1829627340
   [junit4]   2> 2234761 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1982841081
   [junit4]   2> 2234761 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1827606894
   [junit4]   2> 2234981 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2237604 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2237755 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2237756 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2237756 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2237756 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2237760 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=88.177734375, floorSegmentMB=1.3203125, forceMergeDeletesPctAllowed=26.2332621940895, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2239419 WARN  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2239844 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2239861 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2239862 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2239862 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2239894 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2239905 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@aec20bc[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2239907 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2239907 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2239908 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2239909 INFO  (searcherExecutor-2048-thread-1-processing-n:127.0.0.1:37799_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@aec20bc[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2239925 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586376472942084096
   [junit4]   2> 2239962 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2239962 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2239962 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37799/control_collection_shard1_replica_n1/
   [junit4]   2> 2239963 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2239963 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:37799/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2239963 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2240002 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2240002 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37799/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2240003 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2240026 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=8819
   [junit4]   2> 2240090 INFO  (qtp1677638809-3865) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2240277 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2240939 INFO  (OverseerCollectionConfigSetProcessor-99148528425697285-127.0.0.1:37799_-n_0000000000) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2241105 INFO  (qtp1677638809-3865) [n:127.0.0.1:37799_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37799_&wt=javabin&version=2} status=0 QTime=10394
   [junit4]   2> 2241866 INFO  (zkConnectionManagerCallback-676-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2241867 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2241868 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43531/solr ready
   [junit4]   2> 2241868 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2241906 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2241939 INFO  (OverseerThreadFactory-2045-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2241940 WARN  (OverseerThreadFactory-2045-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2242289 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2242289 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=383
   [junit4]   2> 2243982 INFO  (OverseerCollectionConfigSetProcessor-99148528425697285-127.0.0.1:37799_-n_0000000000) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2244242 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001 of type NRT
   [junit4]   2> 2244243 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2244497 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4370e392{/,null,AVAILABLE}
   [junit4]   2> 2245976 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@311e8486{SSL,[ssl, http/1.1]}{127.0.0.1:39353}
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server Started @2263827ms
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=39353, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores}
   [junit4]   2> 2245977 ERROR (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-10T06:15:04.448Z
   [junit4]   2> 2246073 INFO  (zkConnectionManagerCallback-678-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2246113 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2246113 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/solr.xml
   [junit4]   2> 2246116 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2246116 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2246117 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2246585 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43531/solr
   [junit4]   2> 2246788 INFO  (zkConnectionManagerCallback-682-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2247031 INFO  (zkConnectionManagerCallback-684-thread-1-processing-n:127.0.0.1:39353_) [n:127.0.0.1:39353_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2247066 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2247067 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2247086 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39353_
   [junit4]   2> 2247105 INFO  (zkCallback-675-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2247105 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2247453 INFO  (zkCallback-683-thread-1-processing-n:127.0.0.1:39353_) [n:127.0.0.1:39353_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2248384 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2248465 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2248465 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2248529 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores
   [junit4]   2> 2248760 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39353_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2248934 INFO  (OverseerThreadFactory-2045-thread-3-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:39353_ for creating new replica
   [junit4]   2> 2249402 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2249402 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2250557 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2250700 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema [collection1_shard1_replica_n21] Schema name=test
   [junit4]   2> 2253898 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n21' using configuration from collection collection1, trusted=true
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n21' (registry 'solr.core.collection1.shard1.replica_n21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [[collection1_shard1_replica_n21] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores/collection1_shard1_replica_n21], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores/collection1_shard1_replica_n21/data/]
   [junit4]   2> 2254408 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=88.177734375, floorSegmentMB=1.3203125, forceMergeDeletesPctAllowed=26.2332621940895, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2254455 WARN  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2255465 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2255465 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2255466 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2255466 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2255480 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2255490 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.SolrIndexSearcher Opening [Searcher@370f0a01[collection1_shard1_replica_n21] main]
   [junit4]   2> 2255511 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2255512 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2255514 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2255566 INFO  (searcherExecutor-2059-thread-1-processing-n:127.0.0.1:39353_ x:collection1_shard1_replica_n21 s:shard1 c:collection1) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [collection1_shard1_replica_n21] Registered new searcher Searcher@370f0a01[collection1_shard1_replica_n21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2255566 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586376489342861312
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:39353/collection1_shard1_replica_n21/
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy https://127.0.0.1:39353/collection1_shard1_replica_n21/ has no replicas
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2255616 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:39353/collection1_shard1_replica_n21/ shard1
   [junit4]   2> 2255617 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2255635 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=6232
   [junit4]   2> 2255703 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:39353_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=6943
   [junit4]   2> 2257168 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-2-001 of type NRT
   [junit4]   2> 2257206 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2257210 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b954e9a{/,null,AVAILABLE}
   [junit4]   2> 2257211 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@fd85aeb{SSL,[ssl, http/1.1]}{127.0.0.1:60432}
   [junit4]   2> 2257211 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server Started @2275061ms
   [junit4]   2> 2257211 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=60432, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-2-001/cores}
   [junit4]   2> 2257211 ERROR (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-10T06:15:15.729Z
   [junit4]   2> 2257572 INFO  (zkConnectionManagerCallback-687-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2257589 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2257589 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-2-001/solr.xml
   [junit4]   2> 2257592 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2257592 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2257610 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2257614 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43531/solr
   [junit4]   2> 2257823 INFO  (OverseerCollectionConfigSetProcessor-99148528425697285-127.0.0.1:37799_-n_0000000000) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2257838 INFO  (zkConnectionManagerCallback-691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258121 INFO  (zkConnectionManagerCallback-693-thread-1-processing-n:127.0.0.1:60432_) [n:127.0.0.1:60432_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258282 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:60432_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2258283 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:60432_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2258293 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:60432_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60432_
   [junit4]   2> 2258293 INFO  (zkCallback-683-thread-1-processing-n:127.0.0.1:39353_) [n:127.0.0.1:39353_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2258294 INFO  (zkCallback-675-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2258294 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2258434 INFO  (zkCallback-692-thread-1-processing-n:127.0.0.1:60432_) [n:127.0.0.1:

[...truncated too long message...]

OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 9561415 INFO  (jetty-closer-3827-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@ce20312{/solr,null,UNAVAILABLE}
   [junit4]   2> 9561454 INFO  (jetty-closer-3827-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4b1b1608{/solr,null,UNAVAILABLE}
   [junit4]   2> 9561508 INFO  (zkCallback-3858-thread-7-processing-n:127.0.0.1:49189_solr) [n:127.0.0.1:49189_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49189_solr
   [junit4]   2> 9561527 INFO  (zkCallback-3864-thread-10-processing-n:127.0.0.1:34960_solr) [n:127.0.0.1:34960_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34960_solr
   [junit4]   2> 9563013 INFO  (jetty-closer-3827-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@71d6d3ad{/solr,null,UNAVAILABLE}
   [junit4]   2> 9563036 INFO  (jetty-closer-3827-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5ba4cdfa{/solr,null,UNAVAILABLE}
   [junit4]   2> 9563051 ERROR (SUITE-TestTolerantUpdateProcessorCloud-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 9563053 INFO  (SUITE-TestTolerantUpdateProcessorCloud-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57364 57364
   [junit4]   2> 9563171 INFO  (Thread-4195) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57364 57364
   [junit4]   2> 9563177 WARN  (Thread-4195) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 12 /solr/aliases.json
   [junit4]   2> 10 /solr/security.json
   [junit4]   2> 4 /solr/configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 4 /solr/configs/solrCloudCollectionConfig
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 20 /solr/collections/test_col/state.json
   [junit4]   2> 12 /solr/clusterprops.json
   [junit4]   2> 12 /solr/clusterstate.json
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2> 2 /solr/overseer_elect/election/99148996429938706-127.0.0.1:49189_solr-n_0000000006
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 12 /solr/live_nodes
   [junit4]   2> 12 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> 9563208 INFO  (SUITE-TestTolerantUpdateProcessorCloud-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Dec 10, 2017 8:17:04 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=240, maxMBSortInHeap=5.241533852929293, sim=RandomSimilarity(queryNorm=false): {}, locale=mk-MK, timezone=SystemV/EST5
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=306496192,total=525336576
   [junit4]   2> NOTE: All tests run in this JVM: [LukeRequestHandlerTest, SolrIndexSplitterTest, CdcrReplicationHandlerTest, TestStressLucene, TestNestedDocsSort, TestSchemaManager, BlockCacheTest, TestRestoreCore, ClassificationUpdateProcessorIntegrationTest, TlogReplayBufferedWhileIndexingTest, SynonymTokenizerTest, CoreAdminCreateDiscoverTest, TestDocumentBuilder, AutoScalingHandlerTest, SampleTest, MoveReplicaHDFSFailoverTest, JavabinLoaderTest, RegexBoostProcessorTest, TestReplicaProperties, TestFieldCollectionResource, TestJettySolrRunner, TestStressUserVersions, TestEmbeddedSolrServerAdminHandler, BlockJoinFacetSimpleTest, AddReplicaTest, DirectUpdateHandlerTest, HighlighterConfigTest, SolrMetricsIntegrationTest, TestLazyCores, SignatureUpdateProcessorFactoryTest, TestLuceneMatchVersion, DeleteShardTest, TestApiFramework, TestHdfsBackupRestoreCore, DocumentAnalysisRequestHandlerTest, DOMUtilTest, TestFieldCache, TestLeaderElectionZkExpiry, SolrCoreMetricManagerTest, ShardRoutingCustomTest, QueryEqualityTest, SOLR749Test, TestPrepRecovery, SchemaVersionSpecificBehaviorTest, TestSolrCoreSnapshots, DocValuesMissingTest, PrimUtilsTest, TestClassicSimilarityFactory, ChaosMonkeySafeLeaderTest, InfoHandlerTest, DistribDocExpirationUpdateProcessorTest, TestJsonRequest, TestImpersonationWithHadoopAuth, TestCollapseQParserPlugin, HttpSolrCallGetCoreTest, DistributedTermsComponentTest, TestStressRecovery, HdfsDirectoryTest, TestSSLRandomization, OpenExchangeRatesOrgProviderTest, TestHdfsUpdateLog, MBeansHandlerTest, LegacyCloudClusterPropTest, TestSolrCLIRunExample, TestAuthenticationFramework, TestCustomSort, TestQuerySenderNoQuery, PeerSyncReplicationTest, TestPseudoReturnFields, TestCustomDocTransformer, TestUniqueKeyFieldResource, ConcurrentDeleteAndCreateCollectionTest, ImplicitSnitchTest, AtomicUpdateProcessorFactoryTest, TestXmlQParserPlugin, SuggesterTSTTest, PreAnalyzedFieldTest, TestRawResponseWriter, TestSolrCloudWithKerberosAlt, TestNumericRangeQuery32, NumericFieldsTest, TestSizeLimitedDistributedMap, SolrIndexMetricsTest, DistributedIntervalFacetingTest, RequestHandlersTest, TestSurroundQueryParser, TestLegacyFieldCache, TestSimpleTrackingShardHandler, TestBM25SimilarityFactory, TestGraphMLResponseWriter, CreateCollectionCleanupTest, HdfsBasicDistributedZk2Test, TestAnalyzedSuggestions, TestFieldTypeResource, TestBlobHandler, PingRequestHandlerTest, HdfsSyncSliceTest, MultiThreadedOCPTest, TestStressReorder, TestRemoteStreaming, TestStressInPlaceUpdates, SolrCmdDistributorTest, SpellingQueryConverterTest, TestSubQueryTransformerCrossCore, TestLRUStatsCache, SpellCheckCollatorWithCollapseTest, RankQueryTest, SolrCLIZkUtilsTest, TestMaxTokenLenTokenizer, CloudExitableDirectoryReaderTest, OutputWriterTest, CursorMarkTest, TestWriterPerf, SolrRequestParserTest, HdfsAutoAddReplicasIntegrationTest, TestSlowCompositeReaderWrapper, ZkSolrClientTest, BinaryUpdateRequestHandlerTest, TestCursorMarkWithoutUniqueKey, DocValuesMultiTest, LeaderFailureAfterFreshStartTest, CdcrBidirectionalTest, TestUseDocValuesAsStored2, BasicDistributedZkTest, TestCSVLoader, TestAtomicUpdateErrorCases, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestQueryUtils, TestOmitPositions, XmlUpdateRequestHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, TestCollationField, ReturnFieldsTest, TestPHPSerializedResponseWriter, UpdateParamsTest, ResponseLogComponentTest, TestDocSet, TestBinaryField, TestElisionMultitermQuery, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, EchoParamsTest, TestLMDirichletSimilarityFactory, TestFastOutputStream, PluginInfoTest, TestFastLRUCache, DateMathParserTest, TestSolrJ, ZkNodePropsTest, UUIDFieldTest, CircularListTest, TestRTGBase, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, AssignTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, MoveReplicaTest, ReplicationFactorTest, RollingRestartTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCollectionAPI, TestConfigSetsAPI, TestConfigSetsAPIExclusivity, TestDeleteCollectionOnDownNodes, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestRandomRequestDistribution, TestRequestForwarding, TestRequestStatusCollectionAPI, TestSegmentSorting, TestTolerantUpdateProcessorCloud]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTolerantUpdateProcessorCloud -Dtests.seed=5DDE6DD7E5FA5A2A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=mk-MK -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestTolerantUpdateProcessorCloud (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34960/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
   [junit4]    > at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorCloud.java:121)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [559/751 (6!)] on J1 in 203.45s, 0 tests, 1 error <<< FAILURES!

[...truncated 48146 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 280 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/280/

4 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E:1B47F0B7461293A6]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:110)
        at org.apache.solr.cloud.ChaosMonkey.logCollectionStateSummary(ChaosMonkey.java:710)
        at org.apache.solr.cloud.ChaosMonkey.wait(ChaosMonkey.java:704)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:172)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        ... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5913, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=5883, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    8) Thread[id=5882, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    9) Thread[id=5884, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$222/2079815092.execute(Unknown Source)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5913, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=5883, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   8) Thread[id=5882, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   9) Thread[id=5884, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.SolrZkClient$$Lambda$222/2079815092.execute(Unknown Source)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
        at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=5883, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.hashCode(Native Method)         at java.util.WeakHashMap.hash(WeakHashMap.java:298)         at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    7) Thread[id=5882, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.security.AccessController.doPrivileged(Native Method)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.write(PrintStream.java:527)         at java.io.PrintStream.print(PrintStream.java:669)         at java.io.PrintStream.println(PrintStream.java:823)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=5883, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.hashCode(Native Method)
        at java.util.WeakHashMap.hash(WeakHashMap.java:298)
        at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)
        at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
        at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
        at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
        at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   7) Thread[id=5882, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.security.AccessController.doPrivileged(Native Method)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
        at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)


FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":34105},   "errorMessages":["4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",     "code":500}}

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":34105},
  "errorMessages":["4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}

        at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E:1B47F0B7461293A6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:185)
        at org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:253)
        at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:97)
        at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11732 lines...]
   [junit4] Suite: org.apache.solr.handler.TestSolrConfigHandlerCloud
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/init-core-data-001
   [junit4]   2> 360963 WARN  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 360964 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 360991 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 360992 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 360993 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 361036 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 361487 INFO  (Thread-92) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 361487 INFO  (Thread-92) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 361524 ERROR (Thread-92) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 361591 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkTestServer start zk server on port:38165
   [junit4]   2> 361879 INFO  (zkConnectionManagerCallback-113-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 361881 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160467a17a00000, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 361953 INFO  (zkConnectionManagerCallback-115-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 362016 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 362024 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 362032 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 362038 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 362051 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 362052 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 362057 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 362116 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 362117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 362118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 362124 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 363269 INFO  (zkConnectionManagerCallback-118-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 363367 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 365062 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 365116 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79ba0101{/,null,AVAILABLE}
   [junit4]   2> 365117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6a865527{SSL,[ssl, http/1.1]}{127.0.0.1:53547}
   [junit4]   2> 365117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server Started @382705ms
   [junit4]   2> 365117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/tempDir-001/control/data, hostContext=/, hostPort=53547, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores}
   [junit4]   2> 365118 ERROR (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-11T16:48:18.351Z
   [junit4]   2> 365125 INFO  (zkConnectionManagerCallback-120-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 365148 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 365148 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/solr.xml
   [junit4]   2> 366059 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 366059 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 366062 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 366128 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38165/solr
   [junit4]   2> 366353 INFO  (zkConnectionManagerCallback-124-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 366388 INFO  (zkConnectionManagerCallback-126-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 367191 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 367192 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53547_
   [junit4]   2> 367207 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.Overseer Overseer (id=99156681998467077-127.0.0.1:53547_-n_0000000000) starting
   [junit4]   2> 367359 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53547_
   [junit4]   2> 367433 INFO  (zkCallback-125-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 371663 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 371834 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 371834 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 372162 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores
   [junit4]   2> 373323 INFO  (zkConnectionManagerCallback-132-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 373324 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 373399 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38165/solr ready
   [junit4]   2> 374082 INFO  (qtp867239183-513) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:53547_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 374107 INFO  (OverseerThreadFactory-217-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 374584 INFO  (qtp867239183-510) [n:127.0.0.1:53547_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 374584 INFO  (qtp867239183-510) [n:127.0.0.1:53547_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 374744 INFO  (zkCallback-125-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 375884 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 376356 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 378385 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 378659 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 378660 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 378660 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 378660 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 378862 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=920052104, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 378868 WARN  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 379343 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 379343 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 379352 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 379352 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 379407 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=8.2099609375, floorSegmentMB=0.40625, forceMergeDeletesPctAllowed=3.600584705322606, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15346715754849552
   [junit4]   2> 379415 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@39935aaa[control_collection_shard1_replica_n1] main]
   [junit4]   2> 379416 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 379417 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 379417 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 379419 INFO  (searcherExecutor-220-thread-1-processing-n:127.0.0.1:53547_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@39935aaa[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 379423 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586506930749177856
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:53547/control_collection_shard1_replica_n1/
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:53547/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 379531 INFO  (zkCallback-125-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 379543 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:53547/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 379546 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 379575 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=4991
   [junit4]   2> 379620 INFO  (qtp867239183-513) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 379663 INFO  (zkCallback-125-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 380284 INFO  (OverseerCollectionConfigSetProcessor-99156681998467077-127.0.0.1:53547_-n_0000000000) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 380643 INFO  (qtp867239183-513) [n:127.0.0.1:53547_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:53547_&wt=javabin&version=2} status=0 QTime=6561
   [junit4]   2> 380927 INFO  (zkConnectionManagerCallback-136-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 380928 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 380929 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38165/solr ready
   [junit4]   2> 380929 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 381078 INFO  (qtp867239183-511) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 381103 INFO  (OverseerThreadFactory-217-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 381112 WARN  (OverseerThreadFactory-217-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 381252 INFO  (qtp867239183-511) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 381259 INFO  (qtp867239183-511) [n:127.0.0.1:53547_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=181
   [junit4]   2> 381267 INFO  (OverseerCollectionConfigSetProcessor-99156681998467077-127.0.0.1:53547_-n_0000000000) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 384021 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001 of type NRT
   [junit4]   2> 384029 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 384301 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@47dfed71{/,null,AVAILABLE}
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6336bd1c{SSL,[ssl, http/1.1]}{127.0.0.1:38626}
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server Started @401890ms
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=38626, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores}
   [junit4]   2> 384302 ERROR (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.3.0
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-11T16:48:37.535Z
   [junit4]   2> 384479 INFO  (zkConnectionManagerCallback-138-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 384495 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 384495 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/solr.xml
   [junit4]   2> 384532 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 384532 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 384533 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 384620 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38165/solr
   [junit4]   2> 384720 INFO  (zkConnectionManagerCallback-142-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 384880 INFO  (zkConnectionManagerCallback-144-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 384915 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 384957 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 384996 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38626_
   [junit4]   2> 384997 INFO  (zkCallback-125-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 385316 INFO  (zkCallback-135-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 385316 INFO  (zkCallback-143-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 385960 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 386077 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 386077 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 386176 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores
   [junit4]   2> 388185 INFO  (qtp1715011692-565) [n:127.0.0.1:38626_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38626_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 388221 INFO  (OverseerThreadFactory-217-thread-3-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:38626_ for creating new replica
   [junit4]   2> 388533 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 388533 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 388846 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 389471 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 390571 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 390696 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=920052104, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 390717 WARN  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 391235 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 391235 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 391236 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 391236 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 391521 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=8.2099609375, floorSegmentMB=0.40625, forceMergeDeletesPctAllowed=3.600584705322606, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15346715754849552
   [junit4]   2> 391563 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@49f56524[collection1_shard2_replica_n41] main]
   [junit4]   2> 391564 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 391565 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 391566 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 391567 INFO  (searcherExecutor-231-thread-1-processing-n:127.0.0.1:38626_ x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@49f56524[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 391567 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586506943483084800
   [junit4]   2> 391604 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 391604 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38626/collection1_shard2_replica_n41/
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy https://127.0.0.1:38626/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 391622 INFO  (zkCallback-143-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 391623 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38626/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 391639 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 391641 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3108
   [junit4]   2> 391653 INFO  (qtp1715011692-565) [n:127.0.0.1:38626_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:38626_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=3467
   [junit4]   2> 391759 INFO  (zkCallback-143-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 392844 INFO  (OverseerCollectionConfigSetProcessor-99156681998467077-127.0.0.1:53547_-n_0000000000) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 393729 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkin

[...truncated too long message...]

casTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    >    9) Thread[id=5884, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:502)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
   [junit4]    >         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$222/2079815092.execute(Unknown Source)
   [junit4]    >         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
   [junit4]    >         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=5883, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Object.hashCode(Native Method)
   [junit4]    >         at java.util.WeakHashMap.hash(WeakHashMap.java:298)
   [junit4]    >         at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)
   [junit4]    >         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
   [junit4]    >         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    >    7) Thread[id=5882, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4]    >         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4]    >         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4]    >         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:482)
   [junit4]    >         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4]    >         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)