[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 68 - Unstable

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 68 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/68/

1 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:36080/cw_/u/collection1: Server error writing document id 255 to the index

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36080/cw_/u/collection1: Server error writing document id 255 to the index
        at __randomizedtesting.SeedInfo.seed([7953E3FA6BE6649E:F107DC20C51A0966]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:649)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224)
        at org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:576)
        at org.apache.solr.cloud.BasicDistributedZkTest.testUpdateProcessorsRunOnlyOnce(BasicDistributedZkTest.java:746)
        at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:424)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14160 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/init-core-data-001
   [junit4]   2> 2020508 WARN  (SUITE-BasicDistributedZkTest-seed#[7953E3FA6BE6649E]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=52 numCloses=52
   [junit4]   2> 2020508 INFO  (SUITE-BasicDistributedZkTest-seed#[7953E3FA6BE6649E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2020511 INFO  (SUITE-BasicDistributedZkTest-seed#[7953E3FA6BE6649E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2020511 INFO  (SUITE-BasicDistributedZkTest-seed#[7953E3FA6BE6649E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2020511 INFO  (SUITE-BasicDistributedZkTest-seed#[7953E3FA6BE6649E]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /cw_/u
   [junit4]   2> 2020514 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2020515 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2020515 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2020615 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer start zk server on port:45881
   [junit4]   2> 2020615 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45881
   [junit4]   2> 2020615 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 45881
   [junit4]   2> 2020620 INFO  (zkConnectionManagerCallback-7634-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2020623 INFO  (zkConnectionManagerCallback-7636-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2020625 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2020626 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2020628 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2020630 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2020631 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2020632 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2020633 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2020635 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2020636 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2020637 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2020638 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2020640 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2020825 WARN  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2020826 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2020826 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2020826 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2020837 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2020837 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2020837 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2020839 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e283666{/cw_/u,null,AVAILABLE}
   [junit4]   2> 2020841 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c8db494{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:41095}
   [junit4]   2> 2020841 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.s.Server Started @2020954ms
   [junit4]   2> 2020841 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/tempDir-001/control/data, hostContext=/cw_/u, hostPort=41095, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/control-001/cores}
   [junit4]   2> 2020842 ERROR (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2020842 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2020842 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.1.0
   [junit4]   2> 2020842 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2020842 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2020842 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-10T02:18:30.206Z
   [junit4]   2> 2020844 INFO  (zkConnectionManagerCallback-7638-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2020845 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2020846 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/control-001/solr.xml
   [junit4]   2> 2020850 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2020850 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2020852 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2020910 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2020912 WARN  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3187f71b[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2020917 WARN  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1ec82fa[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2020918 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45881/solr
   [junit4]   2> 2020919 INFO  (zkConnectionManagerCallback-7645-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2020922 INFO  (zkConnectionManagerCallback-7647-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2021013 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41095_cw_%2Fu
   [junit4]   2> 2021014 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.Overseer Overseer (id=73106003766804484-127.0.0.1:41095_cw_%2Fu-n_0000000000) starting
   [junit4]   2> 2021026 INFO  (zkConnectionManagerCallback-7654-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2021029 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2021030 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41095_cw_%2Fu
   [junit4]   2> 2021030 INFO  (OverseerStateUpdate-73106003766804484-127.0.0.1:41095_cw_%2Fu-n_0000000000) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:41095_cw_%2Fu
   [junit4]   2> 2021033 INFO  (zkCallback-7653-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2021040 INFO  (OverseerStateUpdate-73106003766804484-127.0.0.1:41095_cw_%2Fu-n_0000000000) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2021060 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2021088 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2021104 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2021104 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2021106 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/control-001/cores
   [junit4]   2> 2021137 INFO  (zkConnectionManagerCallback-7660-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2021138 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2021139 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2021141 INFO  (qtp1572877782-37533) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41095_cw_%252Fu&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2021145 INFO  (OverseerThreadFactory-8188-thread-1-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2021255 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2021256 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2022277 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 2022296 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2022439 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2022459 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2022460 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2022460 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2022464 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1621904239, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2022469 WARN  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2022545 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2022545 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2022547 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2022547 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2022549 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=75.861328125, floorSegmentMB=1.556640625, forceMergeDeletesPctAllowed=3.940884174935909, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=23.39088656989974
   [junit4]   2> 2022550 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@59fb6674[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2022552 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2022552 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2022553 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2022553 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630391723011080192
   [junit4]   2> 2022557 INFO  (searcherExecutor-8193-thread-1-processing-n:127.0.0.1:41095_cw_%2Fu x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@59fb6674[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2022559 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2022560 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 2022564 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2022564 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2022564 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41095/cw_/u/control_collection_shard1_replica_n1/
   [junit4]   2> 2022564 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2022564 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:41095/cw_/u/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2022564 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/73106003766804484-core_node2-n_0000000000
   [junit4]   2> 2022567 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41095/cw_/u/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2022668 INFO  (zkCallback-7646-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2022669 INFO  (zkCallback-7646-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2022670 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2022673 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1418
   [junit4]   2> 2022676 INFO  (qtp1572877782-37533) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2022773 INFO  (zkCallback-7646-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2022773 INFO  (zkCallback-7646-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2022773 INFO  (zkCallback-7646-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2022773 INFO  (qtp1572877782-37533) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41095_cw_%252Fu&wt=javabin&version=2} status=0 QTime=1632
   [junit4]   2> 2022780 INFO  (zkConnectionManagerCallback-7666-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2022782 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2022783 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2022783 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2022784 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2022788 INFO  (OverseerThreadFactory-8188-thread-2-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2022789 INFO  (OverseerCollectionConfigSetProcessor-73106003766804484-127.0.0.1:41095_cw_%2Fu-n_0000000000) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2022993 WARN  (OverseerThreadFactory-8188-thread-2-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2022995 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2022996 INFO  (qtp1572877782-37535) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=211
   [junit4]   2> 2022997 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=4
   [junit4]   2> 2023183 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-1-001 of type NRT
   [junit4]   2> 2023184 WARN  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2023184 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2023184 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2023184 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2023185 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2023185 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2023185 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2023186 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3343586b{/cw_/u,null,AVAILABLE}
   [junit4]   2> 2023186 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@300faead{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:37430}
   [junit4]   2> 2023186 INFO  (closeThreadPool-7667-thread-1) [    ] o.e.j.s.Server Started @2023298ms
   [junit4]   2> 2023186 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/cw_/u, hostPort=37430, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-1-001/cores}
   [junit4]   2> 2023187 ERROR (closeThreadPool-7667-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2023187 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2023187 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.1.0
   [junit4]   2> 2023187 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2023187 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2023187 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-10T02:18:32.551Z
   [junit4]   2> 2023190 INFO  (zkConnectionManagerCallback-7669-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023192 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2023193 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-1-001/solr.xml
   [junit4]   2> 2023197 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2023197 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2023199 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2023366 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-2-001 of type NRT
   [junit4]   2> 2023367 WARN  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2023367 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2023367 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2023367 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2023368 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2023368 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2023368 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2023369 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21a46f50{/cw_/u,null,AVAILABLE}
   [junit4]   2> 2023369 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6c9bb349{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:42367}
   [junit4]   2> 2023369 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.Server Started @2023481ms
   [junit4]   2> 2023369 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/cw_/u, hostPort=42367, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-2-001/cores}
   [junit4]   2> 2023370 ERROR (closeThreadPool-7667-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2023370 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2023370 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.1.0
   [junit4]   2> 2023370 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2023370 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2023370 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-10T02:18:32.734Z
   [junit4]   2> 2023372 INFO  (zkConnectionManagerCallback-7672-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023373 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2023373 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-2-001/solr.xml
   [junit4]   2> 2023378 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2023378 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2023380 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2023534 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2023536 WARN  (closeThreadPool-7667-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@746ab88d[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2023540 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2023543 WARN  (closeThreadPool-7667-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@65643b34[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2023543 WARN  (closeThreadPool-7667-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@76a71084[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2023544 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45881/solr
   [junit4]   2> 2023548 INFO  (zkConnectionManagerCallback-7681-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023550 WARN  (closeThreadPool-7667-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@7bcbe7de[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2023551 INFO  (closeThreadPool-7667-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45881/solr
   [junit4]   2> 2023558 INFO  (zkConnectionManagerCallback-7685-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023563 INFO  (zkConnectionManagerCallback-7687-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023568 INFO  (zkConnectionManagerCallback-7689-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023581 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-3-001 of type NRT
   [junit4]   2> 2023584 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2023585 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2023587 WARN  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2023587 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2023587 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2023587 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2023593 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.ZkController Publish node=127.0.0.1:37430_cw_%2Fu as DOWN
   [junit4]   2> 2023593 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.ZkController Publish node=127.0.0.1:42367_cw_%2Fu as DOWN
   [junit4]   2> 2023595 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2023595 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37430_cw_%2Fu
   [junit4]   2> 2023596 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2023596 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42367_cw_%2Fu
   [junit4]   2> 2023600 INFO  (zkCallback-7646-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2023612 INFO  (zkCallback-7665-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2023616 INFO  (zkCallback-7653-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2023616 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2023616 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2023616 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2023617 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@528e0fba{/cw_/u,null,AVAILABLE}
   [junit4]   2> 2023617 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@76a890ac{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:43553}
   [junit4]   2> 2023617 INFO  (closeThreadPool-7667-thread-3) [    ] o.e.j.s.Server Started @2023729ms
   [junit4]   2> 2023617 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/cw_/u, hostPort=43553, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-3-001/cores}
   [junit4]   2> 2023617 ERROR (closeThreadPool-7667-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2023617 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2023618 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.1.0
   [junit4]   2> 2023618 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2023618 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2023618 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-10T02:18:32.982Z
   [junit4]   2> 2023618 INFO  (zkCallback-7688-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2023618 INFO  (zkCallback-7686-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 2023623 INFO  (zkConnectionManagerCallback-7695-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023628 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2023628 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-3-001/solr.xml
   [junit4]   2> 2023633 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2023633 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2023635 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2023651 INFO  (zkConnectionManagerCallback-7705-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023652 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2023654 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2023655 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2023656 INFO  (zkConnectionManagerCallback-7706-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023663 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2023665 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2023666 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2023722 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2023723 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2023740 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2023740 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2023742 INFO  (closeThreadPool-7667-thread-2) [n:127.0.0.1:42367_cw_%2Fu    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-2-001/cores
   [junit4]   2> 2023751 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2023752 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2023755 INFO  (closeThreadPool-7667-thread-1) [n:127.0.0.1:37430_cw_%2Fu    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-1-001/cores
   [junit4]   2> 2023872 INFO  (TEST-BasicDistributedZkTest.test-seed#[7953E3FA6BE6649E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-4-001 of type NRT
   [junit4]   2> 2023873 WARN  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2023873 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2023873 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2023873 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@497a3816{/cw_/u,null,AVAILABLE}
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@459e632d{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:36080}
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.e.j.s.Server Started @2023988ms
   [junit4]   2> 2023875 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/tempDir-001/jetty4, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/cw_/u, hostPort=36080, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-4-001/cores}
   [junit4]   2> 2023876 ERROR (closeThreadPool-7667-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2023876 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2023876 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.1.0
   [junit4]   2> 2023876 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2023876 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2023876 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-10T02:18:33.240Z
   [junit4]   2> 2023878 INFO  (zkConnectionManagerCallback-7710-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023879 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2023879 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-4-001/solr.xml
   [junit4]   2> 2023884 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2023884 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2023886 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2023911 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2023913 WARN  (closeThreadPool-7667-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3903724d[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2023917 WARN  (closeThreadPool-7667-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@146c804a[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2023918 INFO  (closeThreadPool-7667-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45881/solr
   [junit4]   2> 2023919 INFO  (zkConnectionManagerCallback-7717-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023922 INFO  (zkConnectionManagerCallback-7719-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023927 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2023932 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.ZkController Publish node=127.0.0.1:43553_cw_%2Fu as DOWN
   [junit4]   2> 2023933 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2023933 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43553_cw_%2Fu
   [junit4]   2> 2023934 INFO  (zkCallback-7653-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023934 INFO  (zkCallback-7646-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023934 INFO  (zkCallback-7688-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023940 INFO  (zkCallback-7718-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023941 INFO  (zkCallback-7665-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023941 INFO  (zkCallback-7686-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023941 INFO  (zkCallback-7704-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023941 INFO  (zkCallback-7703-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2023964 INFO  (zkConnectionManagerCallback-7726-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2023965 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2023966 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2023967 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2023996 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2024013 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2024013 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2024015 INFO  (closeThreadPool-7667-thread-3) [n:127.0.0.1:43553_cw_%2Fu    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-3-001/cores
   [junit4]   2> 2024234 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 2024236 WARN  (closeThreadPool-7667-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@71786477[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2024241 WARN  (closeThreadPool-7667-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@2ec7a730[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2024242 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45881/solr
   [junit4]   2> 2024244 INFO  (zkConnectionManagerCallback-7733-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2024251 INFO  (zkConnectionManagerCallback-7735-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2024268 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2024273 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.ZkController Publish node=127.0.0.1:36080_cw_%2Fu as DOWN
   [junit4]   2> 2024276 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2024276 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36080_cw_%2Fu
   [junit4]   2> 2024279 INFO  (zkCallback-7665-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7653-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7703-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7688-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7646-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7704-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7686-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024279 INFO  (zkCallback-7718-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024280 INFO  (zkCallback-7725-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024280 INFO  (zkCallback-7734-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2024311 INFO  (zkConnectionManagerCallback-7742-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2024312 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2024314 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45881/solr ready
   [junit4]   2> 2024314 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2024345 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2024361 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2024362 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2024363 INFO  (closeThreadPool-7667-thread-2) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-4-001/cores
   [junit4]   2> 2024451 INFO  (qtp1551901270-37673) [n:127.0.0.1:36080_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:42367_cw_%252Fu&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2024453 INFO  (qtp1551901270-37672) [n:127.0.0.1:36080_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43553_cw_%252Fu&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2024454 INFO  (qtp1551901270-37671) [n:127.0.0.1:36080_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37430_cw_%252Fu&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2024454 INFO  (qtp1551901270-37675) [n:127.0.0.1:36080_cw_%2Fu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:36080_cw_%252Fu&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2024459 INFO  (OverseerCollectionConfigSetProcessor-73106003766804484-127.0.0.1:41095_cw_%2Fu-n_0000000000) [n:127.0.0.1:41095_cw_%2Fu    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2024465 INFO  (OverseerThreadFactory-8188-thread-4-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:43553_cw_%2Fu for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 2024465 INFO  (OverseerThreadFactory-8188-thread-3-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:42367_cw_%2Fu for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 2024468 INFO  (OverseerThreadFactory-8188-thread-4-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2024469 INFO  (OverseerThreadFactory-8188-thread-3-processing-n:127.0.0.1:41095_cw_%2Fu) [n:127.0.0.1:41095_cw_%2Fu c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2024484 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu    x:collection1_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2024489 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu    x:collection1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2025510 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 2025510 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 2025526 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 2025534 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=test
   [junit4]   2> 2025691 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2025706 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2025718 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 2025718 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2025719 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-3-001/cores/collection1_shard2_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-3-001/cores/collection1_shard2_replica_n1/data/]
   [junit4]   2> 2025741 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1621904239, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2025745 WARN  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2025760 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 2025761 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@137ff20a
   [junit4]   2> 2025762 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-2-001/cores/collection1_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_7953E3FA6BE6649E-001/shard-2-001/cores/collection1_shard1_replica_n2/data/]
   [junit4]   2> 2025766 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1621904239, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2025773 WARN  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2025843 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2025843 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2025847 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2025847 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2025848 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=75.861328125, floorSegmentMB=1.556640625, forceMergeDeletesPctAllowed=3.940884174935909, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=23.39088656989974
   [junit4]   2> 2025850 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5e5fbde8[collection1_shard2_replica_n1] main]
   [junit4]   2> 2025852 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2025854 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2025854 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2025855 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630391726473478144
   [junit4]   2> 2025859 INFO  (searcherExecutor-8234-thread-1-processing-n:127.0.0.1:43553_cw_%2Fu x:collection1_shard2_replica_n1 c:collection1 s:shard2) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@5e5fbde8[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2025865 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2025866 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard2
   [junit4]   2> 2025870 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2025870 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2025870 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43553/cw_/u/collection1_shard2_replica_n1/
   [junit4]   2> 2025871 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2025871 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:43553/cw_/u/collection1_shard2_replica_n1/ has no replicas
   [junit4]   2> 2025871 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard2/leader after winning as /collections/collection1/leader_elect/shard2/election/73106003766804499-core_node3-n_0000000000
   [junit4]   2> 2025874 INFO  (qtp1593382716-37644) [n:127.0.0.1:43553_cw_%2Fu c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43553/cw_/u/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 2025901 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2025901 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2025904 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2025904 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2025905 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=75.861328125, floorSegmentMB=1.556640625, forceMergeDeletesPctAllowed=3.940884174935909, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=23.39088656989974
   [junit4]   2> 2025906 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f480d34[collection1_shard1_replica_n2] main]
   [junit4]   2> 2025907 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2025908 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2025909 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2025909 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630391726530101248
   [junit4]   2> 2025912 INFO  (searcherExecutor-8235-thread-1-processing-n:127.0.0.1:42367_cw_%2Fu x:collection1_shard1_replica_n2 c:collection1 s:shard1) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [collection1_shard1_replica_n2] Registered new searcher Searcher@6f480d34[collection1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2025917 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2025918 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 2025922 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2025923 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2025923 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42367/cw_/u/collection1_shard1_replica_n2/
   [junit4]   2> 2025923 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2025923 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:42367/cw_/u/collection1_shard1_replica_n2/ has no replicas
   [junit4]   2> 2025923 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/73106003766804493-core_node4-n_0000000000
   [junit4]   2> 2025926 INFO  (qtp1209393835-37607) [n:127.0.0.1:42367_cw_%2Fu c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42367/cw_/u/collection1_shard1_replica_n2/ shard1
   [junit4]   2> 2026028 INFO  (zkCallback-7718-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collectio

[...truncated too long message...]



ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null980402615
     [copy] Copying 240 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null980402615
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null980402615

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 74ms :: artifacts dl 9ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 309 minutes 12 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 69 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/69/

6 tests failed.
FAILED:  org.apache.lucene.luke.models.documents.DocumentsImplTest.testGetDocumentFields

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([F480D5057173988C:6B8D3376B872446C]:0)
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertFalse(Assert.java:64)
        at org.junit.Assert.assertFalse(Assert.java:74)
        at org.apache.lucene.luke.models.documents.DocumentsImplTest.testGetDocumentFields(DocumentsImplTest.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper

Error Message:
Didn't get expected doc count. Expected: 10, Found: 0

Stack Trace:
java.lang.AssertionError: Didn't get expected doc count. Expected: 10, Found: 0
        at __randomizedtesting.SeedInfo.seed([310B8A5D0F18826D:2E42AADE2AF28D0D]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.TestPullReplicaErrorHandling.assertNumDocs(TestPullReplicaErrorHandling.java:252)
        at org.apache.solr.cloud.TestPullReplicaErrorHandling.assertNumDocs(TestPullReplicaErrorHandling.java:257)
        at org.apache.solr.cloud.TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper(TestPullReplicaErrorHandling.java:228)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple2 Timeout waiting to see state for collection=testSimple2 :DocCollection(testSimple2//collections/testSimple2/state.json/22)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/",           "base_url":"https://127.0.0.1:43306/solr",           "node_name":"127.0.0.1:43306_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/tlog",           "core":"testSimple2_shard1_replica_n1",           "shared_storage":"true",           "state":"down"},         "core_node5":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/",           "base_url":"https://127.0.0.1:33925/solr",           "node_name":"127.0.0.1:33925_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/tlog",           "core":"testSimple2_shard1_replica_n2",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/",           "base_url":"https://127.0.0.1:43306/solr",           "node_name":"127.0.0.1:43306_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/tlog",           "core":"testSimple2_shard2_replica_n4",           "shared_storage":"true",           "state":"down"},         "core_node8":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/",           "base_url":"https://127.0.0.1:33925/solr",           "node_name":"127.0.0.1:33925_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/tlog",           "core":"testSimple2_shard2_replica_n6",           "shared_storage":"true",           "state":"active",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"} Live Nodes: [127.0.0.1:33925_solr, 127.0.0.1:35154_solr] Last available state: DocCollection(testSimple2//collections/testSimple2/state.json/22)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/",           "base_url":"https://127.0.0.1:43306/solr",           "node_name":"127.0.0.1:43306_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/tlog",           "core":"testSimple2_shard1_replica_n1",           "shared_storage":"true",           "state":"down"},         "core_node5":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/",           "base_url":"https://127.0.0.1:33925/solr",           "node_name":"127.0.0.1:33925_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/tlog",           "core":"testSimple2_shard1_replica_n2",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/",           "base_url":"https://127.0.0.1:43306/solr",           "node_name":"127.0.0.1:43306_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/tlog",           "core":"testSimple2_shard2_replica_n4",           "shared_storage":"true",           "state":"down"},         "core_node8":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/",           "base_url":"https://127.0.0.1:33925/solr",           "node_name":"127.0.0.1:33925_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/tlog",           "core":"testSimple2_shard2_replica_n6",           "shared_storage":"true",           "state":"active",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple2
Timeout waiting to see state for collection=testSimple2 :DocCollection(testSimple2//collections/testSimple2/state.json/22)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/",
          "base_url":"https://127.0.0.1:43306/solr",
          "node_name":"127.0.0.1:43306_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/tlog",
          "core":"testSimple2_shard1_replica_n1",
          "shared_storage":"true",
          "state":"down"},
        "core_node5":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/",
          "base_url":"https://127.0.0.1:33925/solr",
          "node_name":"127.0.0.1:33925_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/tlog",
          "core":"testSimple2_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/",
          "base_url":"https://127.0.0.1:43306/solr",
          "node_name":"127.0.0.1:43306_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/tlog",
          "core":"testSimple2_shard2_replica_n4",
          "shared_storage":"true",
          "state":"down"},
        "core_node8":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/",
          "base_url":"https://127.0.0.1:33925/solr",
          "node_name":"127.0.0.1:33925_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/tlog",
          "core":"testSimple2_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
Live Nodes: [127.0.0.1:33925_solr, 127.0.0.1:35154_solr]
Last available state: DocCollection(testSimple2//collections/testSimple2/state.json/22)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/",
          "base_url":"https://127.0.0.1:43306/solr",
          "node_name":"127.0.0.1:43306_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node3/data/tlog",
          "core":"testSimple2_shard1_replica_n1",
          "shared_storage":"true",
          "state":"down"},
        "core_node5":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/",
          "base_url":"https://127.0.0.1:33925/solr",
          "node_name":"127.0.0.1:33925_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node5/data/tlog",
          "core":"testSimple2_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/",
          "base_url":"https://127.0.0.1:43306/solr",
          "node_name":"127.0.0.1:43306_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node7/data/tlog",
          "core":"testSimple2_shard2_replica_n4",
          "shared_storage":"true",
          "state":"down"},
        "core_node8":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/",
          "base_url":"https://127.0.0.1:33925/solr",
          "node_name":"127.0.0.1:33925_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43289/solr_hdfs_home/testSimple2/core_node8/data/tlog",
          "core":"testSimple2_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([310B8A5D0F18826D:9B8AEA328EB56BC]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:310)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testCreateLargeSimCollections

Error Message:
last ClusterState: znodeVersion: 772 live nodes:[127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr] collections:{.system=DocCollection(.system//clusterstate.json/771)={   "replicationFactor":"3",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{         "core_node1":{           "core":".system_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":8,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":14240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3262033462524414E-5,           "SEARCHER.searcher.numDocs":8},         "core_node2":{           "core":".system_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":8,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":14240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3262033462524414E-5,           "SEARCHER.searcher.numDocs":8},         "core_node3":{           "core":".system_shard1_replica_n3",           "SEARCHER.searcher.maxDoc":8,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":14240,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3262033462524414E-5,           "SEARCHER.searcher.numDocs":8}},       "range":"80000000-7fffffff",       "state":"active"}}}, large_sim_collection2=DocCollection(large_sim_collection2//clusterstate.json/771)={   "replicationFactor":"1",   "pullReplicas":"5",   "router":{"name":"compositeId"},   "maxShardsPerNode":"12",   "autoAddReplicas":"false",   "nrtReplicas":"15",   "tlogReplicas":"19",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node76":{           "core":"large_sim_collection2_shard2_replica_p76",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node54":{           "core":"large_sim_collection2_shard2_replica_n54",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node75":{           "core":"large_sim_collection2_shard2_replica_p75",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node53":{           "core":"large_sim_collection2_shard2_replica_n53",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node56":{           "core":"large_sim_collection2_shard2_replica_t56",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node78":{           "core":"large_sim_collection2_shard2_replica_p78",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node77":{           "core":"large_sim_collection2_shard2_replica_p77",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10005_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node55":{           "core":"large_sim_collection2_shard2_replica_t55",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node58":{           "core":"large_sim_collection2_shard2_replica_t58",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node57":{           "core":"large_sim_collection2_shard2_replica_t57",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node59":{           "core":"large_sim_collection2_shard2_replica_t59",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node61":{           "core":"large_sim_collection2_shard2_replica_t61",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node60":{           "core":"large_sim_collection2_shard2_replica_t60",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node41":{           "core":"large_sim_collection2_shard2_replica_n41",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10021_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node63":{           "core":"large_sim_collection2_shard2_replica_t63",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node62":{           "core":"large_sim_collection2_shard2_replica_t62",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10049_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node40":{           "core":"large_sim_collection2_shard2_replica_n40",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10020_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node43":{           "core":"large_sim_collection2_shard2_replica_n43",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node65":{           "core":"large_sim_collection2_shard2_replica_t65",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node64":{           "core":"large_sim_collection2_shard2_replica_t64",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node42":{           "core":"large_sim_collection2_shard2_replica_n42",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node45":{           "core":"large_sim_collection2_shard2_replica_n45",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node67":{           "core":"large_sim_collection2_shard2_replica_t67",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node66":{           "core":"large_sim_collection2_shard2_replica_t66",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10019_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node44":{           "core":"large_sim_collection2_shard2_replica_n44",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node47":{           "core":"large_sim_collection2_shard2_replica_n47",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node69":{           "core":"large_sim_collection2_shard2_replica_t69",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node46":{           "core":"large_sim_collection2_shard2_replica_n46",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node68":{           "core":"large_sim_collection2_shard2_replica_t68",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node49":{           "core":"large_sim_collection2_shard2_replica_n49",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10095_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node48":{           "core":"large_sim_collection2_shard2_replica_n48",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node70":{           "core":"large_sim_collection2_shard2_replica_t70",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10033_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node72":{           "core":"large_sim_collection2_shard2_replica_t72",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10018_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node50":{           "core":"large_sim_collection2_shard2_replica_n50",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10017_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node71":{           "core":"large_sim_collection2_shard2_replica_t71",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10096_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node52":{           "core":"large_sim_collection2_shard2_replica_n52",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node74":{           "core":"large_sim_collection2_shard2_replica_p74",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node73":{           "core":"large_sim_collection2_shard2_replica_t73",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10035_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node51":{           "core":"large_sim_collection2_shard2_replica_n51",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"aaaa0000-d554ffff",       "state":"active"},     "shard3":{       "replicas":{         "core_node98":{           "core":"large_sim_collection2_shard3_replica_t98",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10065_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node111":{           "core":"large_sim_collection2_shard3_replica_t111",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node110":{           "core":"large_sim_collection2_shard3_replica_t110",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node97":{           "core":"large_sim_collection2_shard3_replica_t97",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node99":{           "core":"large_sim_collection2_shard3_replica_t99",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node79":{           "core":"large_sim_collection2_shard3_replica_n79",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node117":{           "core":"large_sim_collection2_shard3_replica_p117",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node81":{           "core":"large_sim_collection2_shard3_replica_n81",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node80":{           "core":"large_sim_collection2_shard3_replica_n80",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10004_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node116":{           "core":"large_sim_collection2_shard3_replica_p116",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node115":{           "core":"large_sim_collection2_shard3_replica_p115",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10086_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node83":{           "core":"large_sim_collection2_shard3_replica_n83",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node114":{           "core":"large_sim_collection2_shard3_replica_p114",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node82":{           "core":"large_sim_collection2_shard3_replica_n82",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node113":{           "core":"large_sim_collection2_shard3_replica_p113",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10081_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node85":{           "core":"large_sim_collection2_shard3_replica_n85",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node84":{           "core":"large_sim_collection2_shard3_replica_n84",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node112":{           "core":"large_sim_collection2_shard3_replica_t112",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node87":{           "core":"large_sim_collection2_shard3_replica_n87",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10051_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node100":{           "core":"large_sim_collection2_shard3_replica_t100",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node86":{           "core":"large_sim_collection2_shard3_replica_n86",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node89":{           "core":"large_sim_collection2_shard3_replica_n89",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node88":{           "core":"large_sim_collection2_shard3_replica_n88",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node90":{           "core":"large_sim_collection2_shard3_replica_n90",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node108":{           "core":"large_sim_collection2_shard3_replica_t108",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node107":{           "core":"large_sim_collection2_shard3_replica_t107",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10013_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node106":{           "core":"large_sim_collection2_shard3_replica_t106",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node92":{           "core":"large_sim_collection2_shard3_replica_n92",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node91":{           "core":"large_sim_collection2_shard3_replica_n91",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10047_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node105":{           "core":"large_sim_collection2_shard3_replica_t105",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10070_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node94":{           "core":"large_sim_collection2_shard3_replica_t94",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node104":{           "core":"large_sim_collection2_shard3_replica_t104",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node103":{           "core":"large_sim_collection2_shard3_replica_t103",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10066_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node93":{           "core":"large_sim_collection2_shard3_replica_n93",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node102":{           "core":"large_sim_collection2_shard3_replica_t102",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10082_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node96":{           "core":"large_sim_collection2_shard3_replica_t96",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node101":{           "core":"large_sim_collection2_shard3_replica_t101",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10084_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node95":{           "core":"large_sim_collection2_shard3_replica_t95",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node109":{           "core":"large_sim_collection2_shard3_replica_t109",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10063_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"d5550000-ffffffff",       "state":"active"},     "shard4":{       "replicas":{         "core_node133":{           "core":"large_sim_collection2_shard4_replica_t133",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node155":{           "core":"large_sim_collection2_shard4_replica_p155",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node154":{           "core":"large_sim_collection2_shard4_replica_p154",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node132":{           "core":"large_sim_collection2_shard4_replica_n132",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node131":{           "core":"large_sim_collection2_shard4_replica_n131",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10016_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node153":{           "core":"large_sim_collection2_shard4_replica_p153",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node130":{           "core":"large_sim_collection2_shard4_replica_n130",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node152":{           "core":"large_sim_collection2_shard4_replica_p152",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node151":{           "core":"large_sim_collection2_shard4_replica_t151",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node150":{           "core":"large_sim_collection2_shard4_replica_t150",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10017_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node119":{           "core":"large_sim_collection2_shard4_replica_n119",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node118":{           "core":"large_sim_collection2_shard4_replica_n118",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node139":{           "core":"large_sim_collection2_shard4_replica_t139",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node138":{           "core":"large_sim_collection2_shard4_replica_t138",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10053_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node137":{           "core":"large_sim_collection2_shard4_replica_t137",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node136":{           "core":"large_sim_collection2_shard4_replica_t136",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node135":{           "core":"large_sim_collection2_shard4_replica_t135",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node134":{           "core":"large_sim_collection2_shard4_replica_t134",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10022_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node156":{           "core":"large_sim_collection2_shard4_replica_p156",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node144":{           "core":"large_sim_collection2_shard4_replica_t144",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node122":{           "core":"large_sim_collection2_shard4_replica_n122",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node143":{           "core":"large_sim_collection2_shard4_replica_t143",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node121":{           "core":"large_sim_collection2_shard4_replica_n121",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node120":{           "core":"large_sim_collection2_shard4_replica_n120",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node142":{           "core":"large_sim_collection2_shard4_replica_t142",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node141":{           "core":"large_sim_collection2_shard4_replica_t141",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10021_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node140":{           "core":"large_sim_collection2_shard4_replica_t140",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10020_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node129":{           "core":"large_sim_collection2_shard4_replica_n129",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node128":{           "core":"large_sim_collection2_shard4_replica_n128",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node149":{           "core":"large_sim_collection2_shard4_replica_t149",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10095_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node127":{           "core":"large_sim_collection2_shard4_replica_n127",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node148":{           "core":"large_sim_collection2_shard4_replica_t148",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node126":{           "core":"large_sim_collection2_shard4_replica_n126",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node125":{           "core":"large_sim_collection2_shard4_replica_n125",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node147":{           "core":"large_sim_collection2_shard4_replica_t147",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node146":{           "core":"large_sim_collection2_shard4_replica_t146",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node124":{           "core":"large_sim_collection2_shard4_replica_n124",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10072_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node145":{           "core":"large_sim_collection2_shard4_replica_t145",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node123":{           "core":"large_sim_collection2_shard4_replica_n123",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10024_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-2aa9ffff",       "state":"active"},     "shard5":{       "replicas":{         "core_node191":{           "core":"large_sim_collection2_shard5_replica_p191",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10047_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node190":{           "core":"large_sim_collection2_shard5_replica_t190",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node177":{           "core":"large_sim_collection2_shard5_replica_t177",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10005_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node176":{           "core":"large_sim_collection2_shard5_replica_t176",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node175":{           "core":"large_sim_collection2_shard5_replica_t175",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node174":{           "core":"large_sim_collection2_shard5_replica_t174",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node173":{           "core":"large_sim_collection2_shard5_replica_t173",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10035_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node195":{           "core":"large_sim_collection2_shard5_replica_p195",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node172":{           "core":"large_sim_collection2_shard5_replica_t172",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10018_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node194":{           "core":"large_sim_collection2_shard5_replica_p194",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node171":{           "core":"large_sim_collection2_shard5_replica_n171",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10096_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node193":{           "core":"large_sim_collection2_shard5_replica_p193",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node170":{           "core":"large_sim_collection2_shard5_replica_n170",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10033_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node192":{           "core":"large_sim_collection2_shard5_replica_p192",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"PULL",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node159":{           "core":"large_sim_collection2_shard5_replica_n159",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node158":{           "core":"large_sim_collection2_shard5_replica_n158",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node179":{           "core":"large_sim_collection2_shard5_replica_t179",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node157":{           "core":"large_sim_collection2_shard5_replica_n157",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node178":{           "core":"large_sim_collection2_shard5_replica_t178",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node180":{           "core":"large_sim_collection2_shard5_replica_t180",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10004_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node166":{           "core":"large_sim_collection2_shard5_replica_n166",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10019_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node188":{           "core":"large_sim_collection2_shard5_replica_t188",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node187":{           "core":"large_sim_collection2_shard5_replica_t187",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10051_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node165":{           "core":"large_sim_collection2_shard5_replica_n165",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node164":{           "core":"large_sim_collection2_shard5_replica_n164",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node186":{           "core":"large_sim_collection2_shard5_replica_t186",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node163":{           "core":"large_sim_collection2_shard5_replica_n163",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node185":{           "core":"large_sim_collection2_shard5_replica_t185",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node162":{           "core":"large_sim_collection2_shard5_replica_n162",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10049_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node184":{           "core":"large_sim_collection2_shard5_replica_t184",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node161":{           "core":"large_sim_collection2_shard5_replica_n161",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node183":{           "core":"large_sim_collection2_shard5_replica_t183",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node182":{           "core":"large_sim_collection2_shard5_replica_t182",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node160":{           "core":"large_sim_collection2_shard5_replica_n160",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node181":{           "core":"large_sim_collection2_shard5_replica_t181",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node169":{           "core":"large_sim_collection2_shard5_replica_n169",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node168":{           "core":"large_sim_collection2_shard5_replica_n168",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node189":{           "core":"large_sim_collection2_shard5_replica_t189",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"TLOG",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node167":{           "core":"large_sim_collection2_shard5_replica_n167",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"2aaa0000-5554ffff",       "state":"active"},     "shard1":{       "replicas":{         "core_node18":{           "core":"large_sim_collection2_shard1_replica_t18",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,    

[...truncated too long message...]

source] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null204291001
     [copy] Copying 240 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null204291001
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null204291001

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 29ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 581 minutes 28 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 70 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/70/

1 tests failed.
FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([991099BB0F87AB37:AAA2B17F02307180]:0)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:461)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:132)
        at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:116)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:288)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:368)
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
        at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
        at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
        at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
        at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)




Build Log:
[...truncated 13883 lines...]
   [junit4] Suite: org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsUninvertLimit -Dtests.method=testTriggerUnInvertLimit -Dtests.seed=991099BB0F87AB37 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/test-data/enwiki.random.lines.txt -Dtests.locale=mk-MK -Dtests.timezone=Antarctica/DumontDUrville -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    247s J0 | TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([991099BB0F87AB37:AAA2B17F02307180]:0)
   [junit4]    > at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:461)
   [junit4]    > at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:132)
   [junit4]    > at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:116)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:288)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:368)
   [junit4]    > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113)
   [junit4]    > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
   [junit4]    > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
   [junit4]    > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:214)
   [junit4]    > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:106)
   [junit4]    > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
   [junit4]    > at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:68)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.uninverting.TestDocTermOrdsUninvertLimit_991099BB0F87AB37-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {field=PostingsFormat(name=Direct)}, docValues:{}, maxPointsInLeafNode=1502, maxMBSortInHeap=7.244518353878092, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@16748ade), locale=mk-MK, timezone=Antarctica/DumontDUrville
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=68637312,total=534249472
   [junit4]   2> NOTE: All tests run in this JVM: [TestOmitPositions, TestCodecSupport, TestHashPartitioner, TestConfigsApi, IndexSizeTriggerTest, DirectSolrSpellCheckerTest, SolrJmxReporterCloudTest, TestManagedStopFilterFactory, JSONWriterTest, TestStressRecovery, TestChildDocTransformerHierarchy, OverseerStatusTest, UpdateRequestProcessorFactoryTest, SolrIndexMetricsTest, EmbeddedSolrNoSerializeTest, RegexBytesRefFilterTest, ConcurrentDeleteAndCreateCollectionTest, DirectUpdateHandlerTest, ReindexCollectionTest, TimeZoneUtilsTest, MoreLikeThisHandlerTest, TestFieldCacheWithThreads, NodeLostTriggerTest, ResponseHeaderTest, TestCSVResponseWriter, ClassificationUpdateProcessorIntegrationTest, PathHierarchyTokenizerFactoryTest, SolrCloudReportersTest, TestSearcherReuse, TestOrdValues, TestReload, TestUseDocValuesAsStored2, TestSha256AuthenticationProvider, TestDocTermOrdsUninvertLimit]
   [junit4] Completed [169/853 (1!)] on J0 in 246.94s, 1 test, 1 error <<< FAILURES!

[...truncated 2297 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20190411_233336_5512729283246490919842.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/heapdumps/java_pid21475.hprof ...
   [junit4] Heap dump file created [580487893 bytes in 2.723 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 8965 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/build.xml:651: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid21475.hprof

Total time: 291 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 71 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/71/

1 tests failed.
FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
Timeout occurred while waiting response from server at: http://127.0.0.1:39267/_gz/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: http://127.0.0.1:39267/_gz/collection1
        at __randomizedtesting.SeedInfo.seed([DD76BADA1310DA95:55228500BDECB76D]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:660)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.addDocAndGetVersion(TestInPlaceUpdatesDistrib.java:1156)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.buildRandomIndex(TestInPlaceUpdatesDistrib.java:1201)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:351)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:162)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:548)
        ... 49 more




Build Log:
[...truncated 16268 lines...]
   [junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/init-core-data-001
   [junit4]   2> 4371037 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=82 numCloses=82
   [junit4]   2> 4371037 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4371040 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 4371040 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 4371040 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_gz/
   [junit4]   2> 4371040 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 updateLog impl=solr.CdcrUpdateLog
   [junit4]   2> 4371041 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 4371041 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4371074 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 4371088 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
   [junit4]   2> 4371093 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 4371258 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4371260 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@5c24f965[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4371264 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@7d77931d[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4371290 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4371290 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4371312 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4371324 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4371324 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4371330 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4371370 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 4371383 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 4371388 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 4371388 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4371388 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4371388 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/init-core-data-001/]
   [junit4]   2> 4371392 WARN  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4371449 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 4371450 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4371452 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4371452 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4371453 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@56690824[collection1] main]
   [junit4]   2> 4371454 WARN  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 4371454 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4371455 INFO  (coreLoadExecutor-23909-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630676846619131904
   [junit4]   2> 4371457 INFO  (searcherExecutor-23910-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@56690824[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4371458 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[DD76BADA1310DA95]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 4371473 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4371473 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4371474 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4371574 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer start zk server on port:46748
   [junit4]   2> 4371574 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:46748
   [junit4]   2> 4371574 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 46748
   [junit4]   2> 4371584 INFO  (zkConnectionManagerCallback-28653-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4371592 INFO  (zkConnectionManagerCallback-28655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4371594 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 4371596 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
   [junit4]   2> 4371597 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4371598 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 4371600 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 4371601 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 4371602 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 4371604 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4371605 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4371606 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 4371607 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 4371608 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 4371784 WARN  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4371785 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4371785 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4371785 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4371786 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4371786 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4371786 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4371788 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2a7dff32{/_gz,null,AVAILABLE}
   [junit4]   2> 4371788 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6ff51f43{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:44940}
   [junit4]   2> 4371788 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.s.Server Started @4371862ms
   [junit4]   2> 4371788 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/tempDir-001/control/data, hostContext=/_gz, hostPort=44940, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/control-001/cores}
   [junit4]   2> 4371789 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4371789 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4371789 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 4371789 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4371789 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4371789 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-13T05:50:27.313Z
   [junit4]   2> 4371791 INFO  (zkConnectionManagerCallback-28657-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4371792 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4371792 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/control-001/solr.xml
   [junit4]   2> 4371797 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4371797 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4371799 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4372052 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4372053 WARN  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@28099cd7[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4372059 WARN  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@191cc1e0[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4372060 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46748/solr
   [junit4]   2> 4372063 INFO  (zkConnectionManagerCallback-28664-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4372069 INFO  (zkConnectionManagerCallback-28666-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4372164 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44940__gz
   [junit4]   2> 4372165 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.Overseer Overseer (id=73123824070164484-127.0.0.1:44940__gz-n_0000000000) starting
   [junit4]   2> 4372175 INFO  (zkConnectionManagerCallback-28673-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4372177 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46748/solr ready
   [junit4]   2> 4372178 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44940__gz
   [junit4]   2> 4372181 INFO  (zkCallback-28672-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4372181 INFO  (OverseerStateUpdate-73123824070164484-127.0.0.1:44940__gz-n_0000000000) [n:127.0.0.1:44940__gz    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:44940__gz
   [junit4]   2> 4372183 INFO  (OverseerStateUpdate-73123824070164484-127.0.0.1:44940__gz-n_0000000000) [n:127.0.0.1:44940__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4372216 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4372247 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4372264 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4372264 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4372266 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/control-001/cores
   [junit4]   2> 4372312 INFO  (zkConnectionManagerCallback-28679-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4372314 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4372315 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46748/solr ready
   [junit4]   2> 4372317 INFO  (qtp1650460990-86480) [n:127.0.0.1:44940__gz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44940__gz&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4372321 INFO  (OverseerThreadFactory-23924-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 4372429 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4372430 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4373452 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 4373480 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=inplace-updates
   [junit4]   2> 4373500 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 4373500 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 4373501 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4373501 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 4373516 WARN  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4373583 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 4373583 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4373586 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4373586 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4373587 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@d9bdcf4[control_collection_shard1_replica_n1] main]
   [junit4]   2> 4373589 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 4373589 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4373590 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4373590 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630676848857841664
   [junit4]   2> 4373594 INFO  (searcherExecutor-23929-thread-1-processing-n:127.0.0.1:44940__gz x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@d9bdcf4[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4373599 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 4373599 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 4373603 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4373603 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4373603 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44940/_gz/control_collection_shard1_replica_n1/
   [junit4]   2> 4373603 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4373604 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:44940/_gz/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 4373604 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/73123824070164484-core_node2-n_0000000000
   [junit4]   2> 4373606 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44940/_gz/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 4373708 INFO  (zkCallback-28665-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4373708 INFO  (zkCallback-28665-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4373710 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4373713 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1283
   [junit4]   2> 4373729 INFO  (qtp1650460990-86480) [n:127.0.0.1:44940__gz    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 4373813 INFO  (zkCallback-28665-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4373813 INFO  (zkCallback-28665-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4373814 INFO  (qtp1650460990-86480) [n:127.0.0.1:44940__gz    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44940__gz&wt=javabin&version=2} status=0 QTime=1497
   [junit4]   2> 4373818 INFO  (zkCallback-28665-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4373844 INFO  (zkConnectionManagerCallback-28685-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4373852 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4373854 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46748/solr ready
   [junit4]   2> 4373854 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 4373856 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4373860 INFO  (OverseerCollectionConfigSetProcessor-73123824070164484-127.0.0.1:44940__gz-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4373861 INFO  (OverseerThreadFactory-23924-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 4374078 WARN  (OverseerThreadFactory-23924-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 4374080 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 4374081 INFO  (qtp1650460990-86482) [n:127.0.0.1:44940__gz    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=225
   [junit4]   2> 4374082 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 4374302 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-1-001 of type NRT
   [junit4]   2> 4374313 WARN  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4374314 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4374314 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4374314 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4374341 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4374341 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4374341 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4374351 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@73b329b3{/_gz,null,AVAILABLE}
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@319a351d{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:39267}
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.e.j.s.Server Started @4374425ms
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/_gz, hostPort=39267, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-1-001/cores}
   [junit4]   2> 4374352 ERROR (closeThreadPool-28686-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4374352 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-13T05:50:29.876Z
   [junit4]   2> 4374390 INFO  (zkConnectionManagerCallback-28688-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374393 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4374393 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-1-001/solr.xml
   [junit4]   2> 4374398 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4374398 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4374400 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4374574 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-2-001 of type NRT
   [junit4]   2> 4374574 WARN  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4374575 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4374575 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4374575 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4374576 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4374576 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4374576 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4374576 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e6d184a{/_gz,null,AVAILABLE}
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4e554271{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:40585}
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.e.j.s.Server Started @4374650ms
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/_gz, hostPort=40585, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-2-001/cores}
   [junit4]   2> 4374577 ERROR (closeThreadPool-28686-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4374577 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-13T05:50:30.101Z
   [junit4]   2> 4374581 INFO  (zkConnectionManagerCallback-28691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374582 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4374582 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-2-001/solr.xml
   [junit4]   2> 4374587 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4374587 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4374589 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4374616 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4374618 WARN  (closeThreadPool-28686-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@5b8de8aa[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4374622 WARN  (closeThreadPool-28686-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@24c393f0[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4374623 INFO  (closeThreadPool-28686-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46748/solr
   [junit4]   2> 4374625 INFO  (zkConnectionManagerCallback-28698-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374633 INFO  (zkConnectionManagerCallback-28700-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374638 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4374642 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.ZkController Publish node=127.0.0.1:39267__gz as DOWN
   [junit4]   2> 4374643 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4374643 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39267__gz
   [junit4]   2> 4374645 INFO  (zkCallback-28665-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4374646 INFO  (zkCallback-28672-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4374649 INFO  (zkCallback-28699-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4374653 INFO  (zkCallback-28684-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4374693 INFO  (zkConnectionManagerCallback-28707-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374694 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4374696 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46748/solr ready
   [junit4]   2> 4374697 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4374726 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4374746 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4374746 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4374748 INFO  (closeThreadPool-28686-thread-1) [n:127.0.0.1:39267__gz    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-1-001/cores
   [junit4]   2> 4374771 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4374773 WARN  (closeThreadPool-28686-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@53adf73d[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4374778 WARN  (closeThreadPool-28686-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@6a40e599[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4374779 INFO  (closeThreadPool-28686-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46748/solr
   [junit4]   2> 4374782 INFO  (zkConnectionManagerCallback-28714-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374785 INFO  (zkConnectionManagerCallback-28716-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374791 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4374795 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.ZkController Publish node=127.0.0.1:40585__gz as DOWN
   [junit4]   2> 4374796 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4374796 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40585__gz
   [junit4]   2> 4374799 INFO  (zkCallback-28684-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4374799 INFO  (zkCallback-28672-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4374799 INFO  (zkCallback-28665-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4374810 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[DD76BADA1310DA95]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-3-001 of type NRT
   [junit4]   2> 4374813 INFO  (zkCallback-28706-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4374814 WARN  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4374814 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4374814 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4374814 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4374815 INFO  (zkCallback-28699-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4374815 INFO  (zkCallback-28715-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4374816 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4374816 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4374816 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4374817 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a8e3b50{/_gz,null,AVAILABLE}
   [junit4]   2> 4374817 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@40d6e658{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:43420}
   [junit4]   2> 4374817 INFO  (closeThreadPool-28686-thread-3) [    ] o.e.j.s.Server Started @4374890ms
   [junit4]   2> 4374817 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/_gz, hostPort=43420, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-3-001/cores}
   [junit4]   2> 4374817 ERROR (closeThreadPool-28686-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4374817 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4374818 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 4374818 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4374818 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4374818 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-13T05:50:30.342Z
   [junit4]   2> 4374820 INFO  (zkConnectionManagerCallback-28720-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374821 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4374821 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-3-001/solr.xml
   [junit4]   2> 4374826 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4374826 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4374828 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4374835 INFO  (zkConnectionManagerCallback-28726-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4374836 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4374837 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46748/solr ready
   [junit4]   2> 4374838 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4374866 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4374881 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4374881 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4374883 INFO  (closeThreadPool-28686-thread-2) [n:127.0.0.1:40585__gz    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-2-001/cores
   [junit4]   2> 4375041 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 4375045 WARN  (closeThreadPool-28686-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@371ebb70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4375058 WARN  (closeThreadPool-28686-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@25b3303d[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 4375059 INFO  (closeThreadPool-28686-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46748/solr
   [junit4]   2> 4375068 INFO  (zkConnectionManagerCallback-28733-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4375109 INFO  (zkConnectionManagerCallback-28735-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4375115 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4375124 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.ZkController Publish node=127.0.0.1:43420__gz as DOWN
   [junit4]   2> 4375125 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4375125 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43420__gz
   [junit4]   2> 4375127 INFO  (zkCallback-28684-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375128 INFO  (zkCallback-28706-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375128 INFO  (zkCallback-28699-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375128 INFO  (zkCallback-28715-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375128 INFO  (zkCallback-28665-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375129 INFO  (zkCallback-28725-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375133 INFO  (zkCallback-28672-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375136 INFO  (zkCallback-28734-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4375161 INFO  (zkConnectionManagerCallback-28742-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4375166 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4375167 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46748/solr ready
   [junit4]   2> 4375168 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4375196 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4375212 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4375212 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4375214 INFO  (closeThreadPool-28686-thread-3) [n:127.0.0.1:43420__gz    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-3-001/cores
   [junit4]   2> 4375297 INFO  (qtp278150561-86540) [n:127.0.0.1:39267__gz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39267__gz&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4375298 INFO  (qtp278150561-86543) [n:127.0.0.1:39267__gz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40585__gz&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4375299 INFO  (qtp278150561-86539) [n:127.0.0.1:39267__gz    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43420__gz&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4375302 INFO  (OverseerCollectionConfigSetProcessor-73123824070164484-127.0.0.1:44940__gz-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4375310 INFO  (OverseerThreadFactory-23924-thread-3) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:39267__gz for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 4375311 INFO  (OverseerThreadFactory-23924-thread-3) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 4375327 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz    x:collection1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4376347 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 4376361 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=inplace-updates
   [junit4]   2> 4376365 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 4376365 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 4376366 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4376366 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-1-001/cores/collection1_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-1-001/cores/collection1_shard1_replica_n1/data/]
   [junit4]   2> 4376370 WARN  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4376443 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 4376443 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4376446 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4376446 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4376448 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3015e862[collection1_shard1_replica_n1] main]
   [junit4]   2> 4376449 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 4376449 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4376450 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4376450 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630676851856769024
   [junit4]   2> 4376455 INFO  (searcherExecutor-23961-thread-1-processing-n:127.0.0.1:39267__gz x:collection1_shard1_replica_n1 c:collection1 s:shard1) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@3015e862[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4376457 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 4376457 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4376463 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4376463 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4376463 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:39267/_gz/collection1_shard1_replica_n1/
   [junit4]   2> 4376463 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4376464 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:39267/_gz/collection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 4376464 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/73123824070164491-core_node2-n_0000000000
   [junit4]   2> 4376466 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:39267/_gz/collection1_shard1_replica_n1/ shard1
   [junit4]   2> 4376568 INFO  (zkCallback-28699-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4376569 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4376570 INFO  (zkCallback-28699-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4376571 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1244
   [junit4]   2> 4376575 INFO  (qtp278150561-86540) [n:127.0.0.1:39267__gz c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:39267__gz&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1277
   [junit4]   2> 4376673 INFO  (zkCallback-28699-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4376673 INFO  (zkCallback-28699-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4376674 INFO  (zkCallback-28699-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4377307 INFO  (OverseerThreadFactory-23924-thread-4) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40585__gz for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 4377309 INFO  (OverseerCollectionConfigSetProcessor-73123824070164484-127.0.0.1:44940__gz-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4377309 INFO  (OverseerThreadFactory-23924-thread-4) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 4377319 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz    x:collection1_shard1_replica_n3] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4377424 INFO  (zkCallback-28699-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4377424 INFO  (zkCallback-28699-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4377424 INFO  (zkCallback-28699-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4378375 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 4378389 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.IndexSchema [collection1_shard1_replica_n3] Schema name=inplace-updates
   [junit4]   2> 4378393 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 4378393 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n3' using configuration from collection collection1, trusted=true
   [junit4]   2> 4378394 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n3' (registry 'solr.core.collection1.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4309f065
   [junit4]   2> 4378394 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore [[collection1_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-2-001/cores/collection1_shard1_replica_n3], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_DD76BADA1310DA95-001/shard-2-001/cores/collection1_shard1_replica_n3/data/]
   [junit4]   2> 4378407 WARN  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4378488 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 4378488 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4378491 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4378492 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4378493 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@482b2429[collection1_shard1_replica_n3] main]
   [junit4]   2> 4378495 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 4378496 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4378496 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4378497 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1630676854003204096
   [junit4]   2> 4378501 INFO  (searcherExecutor-23966-thread-1-processing-n:127.0.0.1:40585__gz x:collection1_shard1_replica_n3 c:collection1 s:shard1) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore [collection1_shard1_replica_n3] Registered new searcher Searcher@482b2429[collection1_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4378505 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node2=0, core_node4=0}, version=1}
   [junit4]   2> 4378506 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 4378509 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ZkController Core needs to recover:collection1_shard1_replica_n3
   [junit4]   2> 4378509 INFO  (updateExecutor-28710-thread-1-processing-n:127.0.0.1:40585__gz x:collection1_shard1_replica_n3 c:collection1 s:shard1) [n:127.0.0.1:40585__gz c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n3] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 4378509 INFO  (recoveryExecutor-28712-thread-1-processing-n:127.0.0.1:40585__gz x:collection1_shard1_replica_n3 c:collection1 s:shard1 r:core_node4) [n:127.0.0.1:40585__gz c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n3] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 4378515 INFO  (recoveryExecutor-28712-thread-1-processing-n:127.0.0.1:40585__gz x:collection1_shard1_replica_n3 c:collection1 s:shard1 r:core_node4) [n:127.0.0.1:40585__gz c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n3] o.a.s.c.RecoveryStrategy startupVersions is empty
   [junit4]   2> 4378516 INFO  (qtp263276847-86554) [n:127.0.0.1:40585__gz c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1197
   [junit4]   2> 4378518 INFO  (qtp278150561-86541) [n:127.0.0.1:39267__gz c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [coll

[...truncated too long message...]

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null1054849938
     [copy] Copying 240 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null1054849938
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null1054849938

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 39ms :: artifacts dl 2ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 373 minutes 7 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 72 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/72/

No tests ran.

Build Log:
[...truncated 26 lines...]
ERROR: Cannot find executable from the chosen Ant installation "Ant 1.8.4"
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 73 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/73/

4 tests failed.
FAILED:  org.apache.solr.cloud.ReindexCollectionTest.testReshapeReindexing

Error Message:
num docs expected:<200> but was:<198>

Stack Trace:
java.lang.AssertionError: num docs expected:<200> but was:<198>
        at __randomizedtesting.SeedInfo.seed([8F04F081A78B20B3:645983B80890EC46]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.cloud.ReindexCollectionTest.indexDocs(ReindexCollectionTest.java:376)
        at org.apache.solr.cloud.ReindexCollectionTest.testReshapeReindexing(ReindexCollectionTest.java:221)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple2 Timeout waiting to see state for collection=testSimple2 :DocCollection(testSimple2//collections/testSimple2/state.json/23)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/",           "base_url":"https://127.0.0.1:39059/solr",           "node_name":"127.0.0.1:39059_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/tlog",           "core":"testSimple2_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node5":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/",           "base_url":"https://127.0.0.1:43466/solr",           "node_name":"127.0.0.1:43466_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/tlog",           "core":"testSimple2_shard1_replica_n2",           "shared_storage":"true",           "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/",           "base_url":"https://127.0.0.1:39059/solr",           "node_name":"127.0.0.1:39059_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/tlog",           "core":"testSimple2_shard2_replica_n4",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node8":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/",           "base_url":"https://127.0.0.1:43466/solr",           "node_name":"127.0.0.1:43466_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/tlog",           "core":"testSimple2_shard2_replica_n6",           "shared_storage":"true",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"} Live Nodes: [127.0.0.1:34098_solr, 127.0.0.1:39059_solr] Last available state: DocCollection(testSimple2//collections/testSimple2/state.json/23)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/",           "base_url":"https://127.0.0.1:39059/solr",           "node_name":"127.0.0.1:39059_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/tlog",           "core":"testSimple2_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node5":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/",           "base_url":"https://127.0.0.1:43466/solr",           "node_name":"127.0.0.1:43466_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/tlog",           "core":"testSimple2_shard1_replica_n2",           "shared_storage":"true",           "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/",           "base_url":"https://127.0.0.1:39059/solr",           "node_name":"127.0.0.1:39059_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/tlog",           "core":"testSimple2_shard2_replica_n4",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node8":{           "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/",           "base_url":"https://127.0.0.1:43466/solr",           "node_name":"127.0.0.1:43466_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/tlog",           "core":"testSimple2_shard2_replica_n6",           "shared_storage":"true",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple2
Timeout waiting to see state for collection=testSimple2 :DocCollection(testSimple2//collections/testSimple2/state.json/23)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/",
          "base_url":"https://127.0.0.1:39059/solr",
          "node_name":"127.0.0.1:39059_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/tlog",
          "core":"testSimple2_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/",
          "base_url":"https://127.0.0.1:43466/solr",
          "node_name":"127.0.0.1:43466_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/tlog",
          "core":"testSimple2_shard1_replica_n2",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/",
          "base_url":"https://127.0.0.1:39059/solr",
          "node_name":"127.0.0.1:39059_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/tlog",
          "core":"testSimple2_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/",
          "base_url":"https://127.0.0.1:43466/solr",
          "node_name":"127.0.0.1:43466_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/tlog",
          "core":"testSimple2_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
Live Nodes: [127.0.0.1:34098_solr, 127.0.0.1:39059_solr]
Last available state: DocCollection(testSimple2//collections/testSimple2/state.json/23)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/",
          "base_url":"https://127.0.0.1:39059/solr",
          "node_name":"127.0.0.1:39059_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node3/data/tlog",
          "core":"testSimple2_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/",
          "base_url":"https://127.0.0.1:43466/solr",
          "node_name":"127.0.0.1:43466_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node5/data/tlog",
          "core":"testSimple2_shard1_replica_n2",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/",
          "base_url":"https://127.0.0.1:39059/solr",
          "node_name":"127.0.0.1:39059_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node7/data/tlog",
          "core":"testSimple2_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/",
          "base_url":"https://127.0.0.1:43466/solr",
          "node_name":"127.0.0.1:43466_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple2/core_node8/data/tlog",
          "core":"testSimple2_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([8F04F081A78B20B3:B7B7D47F8078F462]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:310)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic

Error Message:
{   "dbLastUpdateTime":1000006030,   "firstTimestamp":1000000020,   "lastTimestamp":1000006080,   "timestamps":[1000000020,1000000080,1000000140,1000000200,1000000260,1000000320,1000000380,1000000440,1000000500,1000000560,1000000620,1000000680,1000000740,1000000800,1000000860,1000000920,1000000980,1000001040,1000001100,1000001160,1000001220,1000001280,1000001340,1000001400,1000001460,1000001520,1000001580,1000001640,1000001700,1000001760,1000001820,1000001880,1000001940,1000002000,1000002060,1000002120,1000002180,1000002240,1000002300,1000002360,1000002420,1000002480,1000002540,1000002600,1000002660,1000002720,1000002780,1000002840,1000002900,1000002960,1000003020,1000003080,1000003140,1000003200,1000003260,1000003320,1000003380,1000003440,1000003500,1000003560,1000003620,1000003680,1000003740,1000003800,1000003860,1000003920,1000003980,1000004040,1000004100,1000004160,1000004220,1000004280,1000004340,1000004400,1000004460,1000004520,1000004580,1000004640,1000004700,1000004760,1000004820,1000004880,1000004940,1000005000,1000005060,1000005120,1000005180,1000005240,1000005300,1000005360,1000005420,1000005480,1000005540,1000005600,1000005660,1000005720,1000005780,1000005840,1000005900,1000005960,1000006020,1000006080],   "data":"1000000020:  +1.0000000000E00  +1.0000000000E02  \n1000000080:  +1.0000000000E00  +1.0000000000E02  \n1000000140:  +1.0000000000E00  +1.0000000000E02  \n1000000200:  +1.0000000000E00  +1.0000000000E02  \n1000000260:  +1.0000000000E00  +1.0000000000E02  \n1000000320:  +1.0000000000E00  +1.0000000000E02  \n1000000380:  +1.0000000000E00  +1.0000000000E02  \n1000000440:  +1.0000000000E00  +1.0000000000E02  \n1000000500:  +1.0000000000E00  +1.0000000000E02  \n1000000560:  +1.0000000000E00  +1.0000000000E02  \n1000000620:  +1.0000000000E00  +1.0000000000E02  \n1000000680:  +1.0000000000E00  +1.0000000000E02  \n1000000740:  +1.0000000000E00  +1.0000000000E02  \n1000000800:  +1.0000000000E00  +1.0000000000E02  \n1000000860:  +1.0000000000E00  +1.0000000000E02  \n1000000920:  +1.0000000000E00  +1.0000000000E02  \n1000000980:  +1.0000000000E00  +1.0000000000E02  \n1000001040:  +1.0000000000E00  +1.0000000000E02  \n1000001100:  +1.0000000000E00  +1.0000000000E02  \n1000001160:  +1.0000000000E00  +1.0000000000E02  \n1000001220:  +1.0000000000E00  +1.0000000000E02  \n1000001280:  +1.0000000000E00  +1.0000000000E02  \n1000001340:  +1.0000000000E00  +1.0000000000E02  \n1000001400:  +1.0000000000E00  +1.0000000000E02  \n1000001460:  +1.0000000000E00  +1.0000000000E02  \n1000001520:  +1.0000000000E00  +1.0000000000E02  \n1000001580:  +1.0000000000E00  +1.0000000000E02  \n1000001640:  +1.0000000000E00  +1.0000000000E02  \n1000001700:  +1.0000000000E00  +1.0000000000E02  \n1000001760:  +1.0000000000E00  +1.0000000000E02  \n1000001820:  +1.0000000000E00  +1.0000000000E02  \n1000001880:  +1.0000000000E00  +1.0000000000E02  \n1000001940:  +1.0000000000E00  +1.0000000000E02  \n1000002000:  +1.0000000000E00  +1.0000000000E02  \n1000002060:  +1.0000000000E00  +1.0000000000E02  \n1000002120:  +1.0000000000E00  +1.0000000000E02  \n1000002180:  +1.0000000000E00  +1.0000000000E02  \n1000002240:  +1.0000000000E00  +1.0000000000E02  \n1000002300:  +1.0000000000E00  +1.0000000000E02  \n1000002360:  +1.0000000000E00  +1.0000000000E02  \n1000002420:  +1.0000000000E00  +1.0000000000E02  \n1000002480:  +1.0000000000E00  +1.0000000000E02  \n1000002540:  +1.0000000000E00  +1.0000000000E02  \n1000002600:  +1.0000000000E00  +1.0000000000E02  \n1000002660:  +1.0000000000E00  +1.0000000000E02  \n1000002720:  +1.0000000000E00  +1.0000000000E02  \n1000002780:  +1.0000000000E00  +1.0000000000E02  \n1000002840:  +1.0000000000E00  +1.0000000000E02  \n1000002900:  +1.0000000000E00  +1.0000000000E02  \n1000002960:  +1.0000000000E00  +1.0000000000E02  \n1000003020:  +1.0000000000E00  +1.0000000000E02  \n1000003080:  +1.0000000000E00  +1.0000000000E02  \n1000003140:  +1.0000000000E00  +1.0000000000E02  \n1000003200:  +1.0000000000E00  +1.0000000000E02  \n1000003260:  +1.0000000000E00  +1.0000000000E02  \n1000003320:  +1.0000000000E00  +1.0000000000E02  \n1000003380:  +1.0000000000E00  +1.0000000000E02  \n1000003440:  +1.0000000000E00  +1.0000000000E02  \n1000003500:  +1.0000000000E00  +1.0000000000E02  \n1000003560:  +1.0000000000E00  +1.0000000000E02  \n1000003620:  +1.0000000000E00  +1.0000000000E02  \n1000003680:  +1.0000000000E00  +1.0000000000E02  \n1000003740:  +1.0000000000E00  +1.0000000000E02  \n1000003800:  +1.0000000000E00  +1.0000000000E02  \n1000003860:  +1.0000000000E00  +1.0000000000E02  \n1000003920:  +1.0000000000E00  +1.0000000000E02  \n1000003980:  +1.0000000000E00  +1.0000000000E02  \n1000004040:  +1.0000000000E00  +1.0000000000E02  \n1000004100:  +1.0000000000E00  +1.0000000000E02  \n1000004160:  +1.0000000000E00  +1.0000000000E02  \n1000004220:  +1.0000000000E00  +1.0000000000E02  \n1000004280:  +1.0000000000E00  +1.0000000000E02  \n1000004340:  +1.0000000000E00  +1.0000000000E02  \n1000004400:  +1.0000000000E00  +1.0000000000E02  \n1000004460:  +1.0000000000E00  +1.0000000000E02  \n1000004520:  +1.0000000000E00  +1.0000000000E02  \n1000004580:  +1.0000000000E00  +1.0000000000E02  \n1000004640:  +1.0000000000E00  +1.0000000000E02  \n1000004700:  +1.0000000000E00  +1.0000000000E02  \n1000004760:  NaN  NaN  \n1000004820:  NaN  NaN  \n1000004880:  NaN  NaN  \n1000004940:  NaN  NaN  \n1000005000:  NaN  NaN  \n1000005060:  NaN  NaN  \n1000005120:  NaN  NaN  \n1000005180:  NaN  NaN  \n1000005240:  NaN  NaN  \n1000005300:  NaN  NaN  \n1000005360:  NaN  NaN  \n1000005420:  NaN  NaN  \n1000005480:  NaN  NaN  \n1000005540:  NaN  NaN  \n1000005600:  NaN  NaN  \n1000005660:  NaN  NaN  \n1000005720:  NaN  NaN  \n1000005780:  NaN  NaN  \n1000005840:  NaN  NaN  \n1000005900:  NaN  NaN  \n1000005960:  NaN  NaN  \n1000006020:  NaN  NaN  \n1000006080:  NaN  NaN  \n"} expected:<NaN> but was:<1.0>

Stack Trace:
java.lang.AssertionError: {
  "dbLastUpdateTime":1000006030,
  "firstTimestamp":1000000020,
  "lastTimestamp":1000006080,
  "timestamps":[1000000020,1000000080,1000000140,1000000200,1000000260,1000000320,1000000380,1000000440,1000000500,1000000560,1000000620,1000000680,1000000740,1000000800,1000000860,1000000920,1000000980,1000001040,1000001100,1000001160,1000001220,1000001280,1000001340,1000001400,1000001460,1000001520,1000001580,1000001640,1000001700,1000001760,1000001820,1000001880,1000001940,1000002000,1000002060,1000002120,1000002180,1000002240,1000002300,1000002360,1000002420,1000002480,1000002540,1000002600,1000002660,1000002720,1000002780,1000002840,1000002900,1000002960,1000003020,1000003080,1000003140,1000003200,1000003260,1000003320,1000003380,1000003440,1000003500,1000003560,1000003620,1000003680,1000003740,1000003800,1000003860,1000003920,1000003980,1000004040,1000004100,1000004160,1000004220,1000004280,1000004340,1000004400,1000004460,1000004520,1000004580,1000004640,1000004700,1000004760,1000004820,1000004880,1000004940,1000005000,1000005060,1000005120,1000005180,1000005240,1000005300,1000005360,1000005420,1000005480,1000005540,1000005600,1000005660,1000005720,1000005780,1000005840,1000005900,1000005960,1000006020,1000006080],
  "data":"1000000020:  +1.0000000000E00  +1.0000000000E02  \n1000000080:  +1.0000000000E00  +1.0000000000E02  \n1000000140:  +1.0000000000E00  +1.0000000000E02  \n1000000200:  +1.0000000000E00  +1.0000000000E02  \n1000000260:  +1.0000000000E00  +1.0000000000E02  \n1000000320:  +1.0000000000E00  +1.0000000000E02  \n1000000380:  +1.0000000000E00  +1.0000000000E02  \n1000000440:  +1.0000000000E00  +1.0000000000E02  \n1000000500:  +1.0000000000E00  +1.0000000000E02  \n1000000560:  +1.0000000000E00  +1.0000000000E02  \n1000000620:  +1.0000000000E00  +1.0000000000E02  \n1000000680:  +1.0000000000E00  +1.0000000000E02  \n1000000740:  +1.0000000000E00  +1.0000000000E02  \n1000000800:  +1.0000000000E00  +1.0000000000E02  \n1000000860:  +1.0000000000E00  +1.0000000000E02  \n1000000920:  +1.0000000000E00  +1.0000000000E02  \n1000000980:  +1.0000000000E00  +1.0000000000E02  \n1000001040:  +1.0000000000E00  +1.0000000000E02  \n1000001100:  +1.0000000000E00  +1.0000000000E02  \n1000001160:  +1.0000000000E00  +1.0000000000E02  \n1000001220:  +1.0000000000E00  +1.0000000000E02  \n1000001280:  +1.0000000000E00  +1.0000000000E02  \n1000001340:  +1.0000000000E00  +1.0000000000E02  \n1000001400:  +1.0000000000E00  +1.0000000000E02  \n1000001460:  +1.0000000000E00  +1.0000000000E02  \n1000001520:  +1.0000000000E00  +1.0000000000E02  \n1000001580:  +1.0000000000E00  +1.0000000000E02  \n1000001640:  +1.0000000000E00  +1.0000000000E02  \n1000001700:  +1.0000000000E00  +1.0000000000E02  \n1000001760:  +1.0000000000E00  +1.0000000000E02  \n1000001820:  +1.0000000000E00  +1.0000000000E02  \n1000001880:  +1.0000000000E00  +1.0000000000E02  \n1000001940:  +1.0000000000E00  +1.0000000000E02  \n1000002000:  +1.0000000000E00  +1.0000000000E02  \n1000002060:  +1.0000000000E00  +1.0000000000E02  \n1000002120:  +1.0000000000E00  +1.0000000000E02  \n1000002180:  +1.0000000000E00  +1.0000000000E02  \n1000002240:  +1.0000000000E00  +1.0000000000E02  \n1000002300:  +1.0000000000E00  +1.0000000000E02  \n1000002360:  +1.0000000000E00  +1.0000000000E02  \n1000002420:  +1.0000000000E00  +1.0000000000E02  \n1000002480:  +1.0000000000E00  +1.0000000000E02  \n1000002540:  +1.0000000000E00  +1.0000000000E02  \n1000002600:  +1.0000000000E00  +1.0000000000E02  \n1000002660:  +1.0000000000E00  +1.0000000000E02  \n1000002720:  +1.0000000000E00  +1.0000000000E02  \n1000002780:  +1.0000000000E00  +1.0000000000E02  \n1000002840:  +1.0000000000E00  +1.0000000000E02  \n1000002900:  +1.0000000000E00  +1.0000000000E02  \n1000002960:  +1.0000000000E00  +1.0000000000E02  \n1000003020:  +1.0000000000E00  +1.0000000000E02  \n1000003080:  +1.0000000000E00  +1.0000000000E02  \n1000003140:  +1.0000000000E00  +1.0000000000E02  \n1000003200:  +1.0000000000E00  +1.0000000000E02  \n1000003260:  +1.0000000000E00  +1.0000000000E02  \n1000003320:  +1.0000000000E00  +1.0000000000E02  \n1000003380:  +1.0000000000E00  +1.0000000000E02  \n1000003440:  +1.0000000000E00  +1.0000000000E02  \n1000003500:  +1.0000000000E00  +1.0000000000E02  \n1000003560:  +1.0000000000E00  +1.0000000000E02  \n1000003620:  +1.0000000000E00  +1.0000000000E02  \n1000003680:  +1.0000000000E00  +1.0000000000E02  \n1000003740:  +1.0000000000E00  +1.0000000000E02  \n1000003800:  +1.0000000000E00  +1.0000000000E02  \n1000003860:  +1.0000000000E00  +1.0000000000E02  \n1000003920:  +1.0000000000E00  +1.0000000000E02  \n1000003980:  +1.0000000000E00  +1.0000000000E02  \n1000004040:  +1.0000000000E00  +1.0000000000E02  \n1000004100:  +1.0000000000E00  +1.0000000000E02  \n1000004160:  +1.0000000000E00  +1.0000000000E02  \n1000004220:  +1.0000000000E00  +1.0000000000E02  \n1000004280:  +1.0000000000E00  +1.0000000000E02  \n1000004340:  +1.0000000000E00  +1.0000000000E02  \n1000004400:  +1.0000000000E00  +1.0000000000E02  \n1000004460:  +1.0000000000E00  +1.0000000000E02  \n1000004520:  +1.0000000000E00  +1.0000000000E02  \n1000004580:  +1.0000000000E00  +1.0000000000E02  \n1000004640:  +1.0000000000E00  +1.0000000000E02  \n1000004700:  +1.0000000000E00  +1.0000000000E02  \n1000004760:  NaN  NaN  \n1000004820:  NaN  NaN  \n1000004880:  NaN  NaN  \n1000004940:  NaN  NaN  \n1000005000:  NaN  NaN  \n1000005060:  NaN  NaN  \n1000005120:  NaN  NaN  \n1000005180:  NaN  NaN  \n1000005240:  NaN  NaN  \n1000005300:  NaN  NaN  \n1000005360:  NaN  NaN  \n1000005420:  NaN  NaN  \n1000005480:  NaN  NaN  \n1000005540:  NaN  NaN  \n1000005600:  NaN  NaN  \n1000005660:  NaN  NaN  \n1000005720:  NaN  NaN  \n1000005780:  NaN  NaN  \n1000005840:  NaN  NaN  \n1000005900:  NaN  NaN  \n1000005960:  NaN  NaN  \n1000006020:  NaN  NaN  \n1000006080:  NaN  NaN  \n"} expected:<NaN> but was:<1.0>
        at __randomizedtesting.SeedInfo.seed([8F04F081A78B20B3:24FEED947857A69D]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:553)
        at org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic(SolrRrdBackendFactoryTest.java:160)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.security.AuditLoggerIntegrationTest.testAsyncQueueDrain

Error Message:
Failed waiting for 3 callbacks after 5 seconds

Stack Trace:
java.lang.AssertionError: Failed waiting for 3 callbacks after 5 seconds
        at __randomizedtesting.SeedInfo.seed([8F04F081A78B20B3:3E1B0FA51748B36C]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.security.AuditLoggerIntegrationTest.waitForAuditEventCallbacks(AuditLoggerIntegrationTest.java:229)
        at org.apache.solr.security.AuditLoggerIntegrationTest.waitForAuditEventCallbacks(AuditLoggerIntegrationTest.java:221)
        at org.apache.solr.security.AuditLoggerIntegrationTest.assertThreeAdminEvents(AuditLoggerIntegrationTest.java:252)
        at org.apache.solr.security.AuditLoggerIntegrationTest.testAsyncQueueDrain(AuditLoggerIntegrationTest.java:132)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14716 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
   [junit4]   2> 3871128 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/init-core-data-001
   [junit4]   2> 3871128 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=93 numCloses=93
   [junit4]   2> 3871128 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3871130 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3871271 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3871304 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3871305 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3871376 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3871376 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3871376 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3871384 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@256f4ed1{static,/static,jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
   [junit4]   2> 3871630 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@606034f5{hdfs,/,file:///home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/jetty-lucene2-us-west.apache.org-34078-hdfs-_-any-3544490536156617045.dir/webapp/,AVAILABLE}{/hdfs}
   [junit4]   2> 3871632 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4373f059{HTTP/1.1,[http/1.1]}{lucene2-us-west.apache.org:34078}
   [junit4]   2> 3871632 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.Server Started @3871717ms
   [junit4]   2> 3872056 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3872057 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3872061 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3872061 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3872061 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3872062 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69b3f9e2{static,/static,jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
   [junit4]   2> 3872304 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@c45c21e{datanode,/,file:///home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-34484-datanode-_-any-4962448433812154357.dir/webapp/,AVAILABLE}{/datanode}
   [junit4]   2> 3872305 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5a329e16{HTTP/1.1,[http/1.1]}{localhost:34484}
   [junit4]   2> 3872305 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.Server Started @3872390ms
   [junit4]   2> 3872520 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3872521 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3872544 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3872544 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3872544 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3872556 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@57f32614{static,/static,jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
   [junit4]   2> 3872734 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xff5d846cbb5099a: Processing first storage report for DS-bd247ffd-2337-4497-bb50-b670042f3753 from datanode 26bccc33-f0c4-419a-985f-43b7e9150c3d
   [junit4]   2> 3872734 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xff5d846cbb5099a: from storage DS-bd247ffd-2337-4497-bb50-b670042f3753 node DatanodeRegistration(127.0.0.1:35045, datanodeUuid=26bccc33-f0c4-419a-985f-43b7e9150c3d, infoPort=38902, infoSecurePort=0, ipcPort=43125, storageInfo=lv=-57;cid=testClusterID;nsid=48078687;c=1555479608409), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
   [junit4]   2> 3872734 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xff5d846cbb5099a: Processing first storage report for DS-1d1e30e2-4863-40b4-905f-068bd1ef2e9e from datanode 26bccc33-f0c4-419a-985f-43b7e9150c3d
   [junit4]   2> 3872734 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xff5d846cbb5099a: from storage DS-1d1e30e2-4863-40b4-905f-068bd1ef2e9e node DatanodeRegistration(127.0.0.1:35045, datanodeUuid=26bccc33-f0c4-419a-985f-43b7e9150c3d, infoPort=38902, infoSecurePort=0, ipcPort=43125, storageInfo=lv=-57;cid=testClusterID;nsid=48078687;c=1555479608409), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
   [junit4]   2> 3872791 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@471f277e{datanode,/,file:///home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-44976-datanode-_-any-2257930517096102312.dir/webapp/,AVAILABLE}{/datanode}
   [junit4]   2> 3872791 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33a31728{HTTP/1.1,[http/1.1]}{localhost:44976}
   [junit4]   2> 3872791 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[8F04F081A78B20B3]-worker) [    ] o.e.j.s.Server Started @3872876ms
   [junit4]   2> 3873101 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcf80ec23483abff7: Processing first storage report for DS-cd259ff0-7480-4846-9fe5-67b860623a7f from datanode 86d3f2aa-b607-467c-8f08-0a91f217ab62
   [junit4]   2> 3873101 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcf80ec23483abff7: from storage DS-cd259ff0-7480-4846-9fe5-67b860623a7f node DatanodeRegistration(127.0.0.1:34341, datanodeUuid=86d3f2aa-b607-467c-8f08-0a91f217ab62, infoPort=44602, infoSecurePort=0, ipcPort=38695, storageInfo=lv=-57;cid=testClusterID;nsid=48078687;c=1555479608409), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
   [junit4]   2> 3873101 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcf80ec23483abff7: Processing first storage report for DS-7a46ea7f-db72-40d4-aa20-ba8e09e8a34a from datanode 86d3f2aa-b607-467c-8f08-0a91f217ab62
   [junit4]   2> 3873101 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xcf80ec23483abff7: from storage DS-7a46ea7f-db72-40d4-aa20-ba8e09e8a34a node DatanodeRegistration(127.0.0.1:34341, datanodeUuid=86d3f2aa-b607-467c-8f08-0a91f217ab62, infoPort=44602, infoSecurePort=0, ipcPort=38695, storageInfo=lv=-57;cid=testClusterID;nsid=48078687;c=1555479608409), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
   [junit4]   2> 3873280 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
   [junit4]   2> 3873281 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002
   [junit4]   2> 3873282 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3873295 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3873295 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3873392 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.ZkTestServer start zk server on port:43830
   [junit4]   2> 3873392 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43830
   [junit4]   2> 3873392 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43830
   [junit4]   2> 3873441 INFO  (zkConnectionManagerCallback-6563-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3873482 INFO  (zkConnectionManagerCallback-6565-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3873518 INFO  (zkConnectionManagerCallback-6567-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3873525 WARN  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 3873525 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 3873525 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 3873525 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3873537 WARN  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 3873537 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 3873537 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 3873537 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3873539 WARN  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 3873539 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 3873539 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 3873539 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3873624 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3873624 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3873624 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3873624 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7135aa5c{/solr,null,AVAILABLE}
   [junit4]   2> 3873625 WARN  (jetty-launcher-6568-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@76d23b44[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3873684 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3873684 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3873684 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3873693 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@26620142{/solr,null,AVAILABLE}
   [junit4]   2> 3873694 WARN  (jetty-launcher-6568-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3dba19c9[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c9adc18{SSL,[ssl, http/1.1]}{127.0.0.1:43466}
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.e.j.s.Server Started @3873779ms
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43466}
   [junit4]   2> 3873694 ERROR (jetty-launcher-6568-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3873694 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-17T05:40:10.885Z
   [junit4]   2> 3873715 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3873715 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3873715 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3873715 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2c308a52{/solr,null,AVAILABLE}
   [junit4]   2> 3873716 WARN  (jetty-launcher-6568-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1e1ffd57[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3aadea5f{SSL,[ssl, http/1.1]}{127.0.0.1:39059}
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.e.j.s.Server Started @3873803ms
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39059}
   [junit4]   2> 3873718 ERROR (jetty-launcher-6568-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3873718 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-17T05:40:10.910Z
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7b5c79c2{SSL,[ssl, http/1.1]}{127.0.0.1:34098}
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.e.j.s.Server Started @3873804ms
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34098}
   [junit4]   2> 3873719 ERROR (jetty-launcher-6568-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.1.0
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3873719 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-04-17T05:40:10.910Z
   [junit4]   2> 3873721 INFO  (zkConnectionManagerCallback-6572-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3873721 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3873725 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3873725 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3873726 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3873732 INFO  (zkConnectionManagerCallback-6570-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3873732 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3873734 INFO  (zkConnectionManagerCallback-6574-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3873735 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3873735 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3873736 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3873739 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3873742 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3873742 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3873743 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3873883 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 3873883 WARN  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2
   [junit4]   2> 3873904 WARN  (jetty-launcher-6568-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@6d811f94[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3873916 WARN  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2
   [junit4]   2> 3873996 WARN  (jetty-launcher-6568-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@6645e9fa[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3874009 INFO  (jetty-launcher-6568-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43830/solr
   [junit4]   2> 3874014 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 3874015 WARN  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2
   [junit4]   2> 3874056 INFO  (zkConnectionManagerCallback-6583-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874066 WARN  (jetty-launcher-6568-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@724d04d8[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3874077 INFO  (zkConnectionManagerCallback-6587-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874095 WARN  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2
   [junit4]   2> 3874101 WARN  (jetty-launcher-6568-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@564695d4[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3874102 INFO  (jetty-launcher-6568-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43830/solr
   [junit4]   2> 3874137 INFO  (zkConnectionManagerCallback-6593-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874138 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x108ce11cce90008, likely client has closed socket
   [junit4]   2> 3874138 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34098_solr
   [junit4]   2> 3874140 INFO  (zkConnectionManagerCallback-6595-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874143 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.Overseer Overseer (id=74535969699201031-127.0.0.1:34098_solr-n_0000000000) starting
   [junit4]   2> 3874145 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43466_solr
   [junit4]   2> 3874146 INFO  (zkCallback-6586-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3874162 INFO  (zkCallback-6594-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3874193 INFO  (zkConnectionManagerCallback-6602-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874194 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3874195 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43830/solr ready
   [junit4]   2> 3874197 INFO  (zkConnectionManagerCallback-6607-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874198 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3874199 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43830/solr ready
   [junit4]   2> 3874201 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:34098_solr as DOWN
   [junit4]   2> 3874201 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3874201 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34098_solr
   [junit4]   2> 3874220 INFO  (OverseerStateUpdate-74535969699201031-127.0.0.1:34098_solr-n_0000000000) [n:127.0.0.1:34098_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:34098_solr
   [junit4]   2> 3874226 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3874226 INFO  (OverseerStateUpdate-74535969699201031-127.0.0.1:34098_solr-n_0000000000) [n:127.0.0.1:34098_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3874239 INFO  (zkCallback-6594-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3874239 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3874241 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874255 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874255 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874256 INFO  (zkCallback-6606-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3874258 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874263 INFO  (jetty-launcher-6568-thread-2) [n:127.0.0.1:43466_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002/node2/.
   [junit4]   2> 3874266 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874266 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874267 INFO  (jetty-launcher-6568-thread-3) [n:127.0.0.1:34098_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002/node3/.
   [junit4]   2> 3874273 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 3874274 WARN  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2
   [junit4]   2> 3874279 INFO  (zkCallback-6601-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3874287 WARN  (jetty-launcher-6568-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@23f99507[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3874303 WARN  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2
   [junit4]   2> 3874312 WARN  (jetty-launcher-6568-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@52413fad[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3874313 INFO  (jetty-launcher-6568-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43830/solr
   [junit4]   2> 3874343 INFO  (zkConnectionManagerCallback-6615-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874364 INFO  (zkConnectionManagerCallback-6617-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874368 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3874371 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:39059_solr as DOWN
   [junit4]   2> 3874371 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3874371 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39059_solr
   [junit4]   2> 3874373 INFO  (zkCallback-6601-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3874373 INFO  (zkCallback-6606-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3874373 INFO  (zkCallback-6594-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3874374 INFO  (zkCallback-6586-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3874380 INFO  (zkCallback-6616-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3874445 INFO  (zkConnectionManagerCallback-6624-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874446 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3874447 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43830/solr ready
   [junit4]   2> 3874458 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3874478 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874529 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874529 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3874530 INFO  (jetty-launcher-6568-thread-1) [n:127.0.0.1:39059_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002/node1/.
   [junit4]   2> 3874597 INFO  (zkConnectionManagerCallback-6630-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3874598 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 3874600 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[8F04F081A78B20B3]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43830/solr ready
   [junit4]   2> 3874634 INFO  (qtp1437645320-20915) [n:127.0.0.1:43466_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster params={wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 3874695 INFO  (qtp1772221782-20927) [n:127.0.0.1:39059_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:34098_solr,127.0.0.1:43466_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3874708 INFO  (OverseerThreadFactory-6643-thread-1-processing-n:127.0.0.1:34098_solr) [n:127.0.0.1:34098_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
   [junit4]   2> 3874817 INFO  (OverseerStateUpdate-74535969699201031-127.0.0.1:34098_solr-n_0000000000) [n:127.0.0.1:34098_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34098/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3874820 INFO  (OverseerStateUpdate-74535969699201031-127.0.0.1:34098_solr-n_0000000000) [n:127.0.0.1:34098_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:43466/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3874822 INFO  (OverseerStateUpdate-74535969699201031-127.0.0.1:34098_solr-n_0000000000) [n:127.0.0.1:34098_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34098/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3874824 INFO  (OverseerStateUpdate-74535969699201031-127.0.0.1:34098_solr-n_0000000000) [n:127.0.0.1:34098_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n7",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:43466/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 3875064 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr    x:testSimple1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3875065 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr    x:testSimple1_shard1_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3875071 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr    x:testSimple1_shard2_replica_n7] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n7&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3875080 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr    x:testSimple1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3875084 INFO  (qtp717794436-20930) [n:127.0.0.1:34098_solr    x:testSimple1_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3876080 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 3876087 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 3876098 INFO  (qtp717794436-20930) [n:127.0.0.1:34098_solr c:testSimple1 s:shard2 r:core_node6 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 3876101 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 3876103 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.1.0
   [junit4]   2> 3876140 INFO  (qtp717794436-20930) [n:127.0.0.1:34098_solr c:testSimple1 s:shard2 r:core_node6 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 3876141 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3876141 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n2' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3876141 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard1.replica_n2' (registry 'solr.core.testSimple1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3876141 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home
   [junit4]   2> 3876142 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3876142 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002/node2/testSimple1_shard1_replica_n2], dataDir=[hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple1/core_node5/data/]
   [junit4]   2> 3876142 INFO  (qtp717794436-20930) [n:127.0.0.1:34098_solr c:testSimple1 s:shard2 r:core_node6 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3876142 INFO  (qtp717794436-20930) [n:127.0.0.1:34098_solr c:testSimple1 s:shard2 r:core_node6 x:testSimple1_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n4' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3876142 INFO  (qtp1437645320-20918) [n:127.0.0.1:43466_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple1/core_node5/data/snapshot_metadata
   [junit4]   2> 3876142 INFO  (qtp717794436-20930) [n:127.0.0.1:34098_solr c:testSimple1 s:shard2 r:core_node6 x:testSimple1_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard2.replica_n4' (registry 'solr.core.testSimple1.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3876144 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 3876145 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n7] Schema name=minimal
   [junit4]   2> 3876146 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3876146 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n1' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3876146 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard1.replica_n1' (registry 'solr.core.testSimple1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3876146 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home
   [junit4]   2> 3876146 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3876146 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002/node3/testSimple1_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple1/core_node3/data/]
   [junit4]   2> 3876147 INFO  (qtp717794436-20932) [n:127.0.0.1:34098_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple1/core_node3/data/snapshot_metadata
   [junit4]   2> 3876147 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3876147 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n7' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 3876147 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard2.replica_n7' (registry 'solr.core.testSimple1.shard2.replica_n7') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@50ff85a1
   [junit4]   2> 3876148 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home
   [junit4]   2> 3876148 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3876148 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n7] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_8F04F081A78B20B3-001/tempDir-002/node2/testSimple1_shard2_replica_n7], dataDir=[hdfs://lucene2-us-west.apache.org:38152/solr_hdfs_home/testSimple1/core_node8/data/]
   [junit4]   2> 3876149 INFO  (qtp1437645320-20916) [n:127.0.0.1:43466_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n7] o.a.s.c.HdfsDi

[...truncated too long message...]

rce] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null1867286220
     [copy] Copying 240 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null1867286220
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.x/checkout/solr/null1867286220

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 63ms :: artifacts dl 8ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 597 minutes 40 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]