[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1345 - Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1345 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1345/

1 tests failed.
FAILED:  org.apache.solr.schema.TestUseDocValuesAsStored.testMultipleSearchResults

Error Message:
mismatch: 'myid1'!='myid3' @ response/docs/[0]/id

Stack Trace:
java.lang.RuntimeException: mismatch: 'myid1'!='myid3' @ response/docs/[0]/id
        at __randomizedtesting.SeedInfo.seed([DD14891C7B505624:EF3E8E8283AE72FD]:0)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:966)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:913)
        at org.apache.solr.schema.TestUseDocValuesAsStored.testMultipleSearchResults(TestUseDocValuesAsStored.java:243)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13503 lines...]
   [junit4] Suite: org.apache.solr.schema.TestUseDocValuesAsStored
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/init-core-data-001
   [junit4]   2> 4880717 WARN  (SUITE-TestUseDocValuesAsStored-seed#[DD14891C7B505624]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 4880718 INFO  (SUITE-TestUseDocValuesAsStored-seed#[DD14891C7B505624]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4880719 INFO  (SUITE-TestUseDocValuesAsStored-seed#[DD14891C7B505624]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 4880723 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testManagedSchema
   [junit4]   2> 4880726 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 4880770 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4880779 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema-non-stored-docvalues.xml instead
   [junit4]   2> 4880781 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.IndexSchema [null] Schema name=test-non-stored-docvalues
   [junit4]   2> 4880788 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.IndexSchema Loaded schema test-non-stored-docvalues/1.6 with uniqueid field id
   [junit4]   2> 4880788 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchema Upgraded to managed schema at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1/conf/managed-schema
   [junit4]   2> 4880798 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema, renamed the non-managed schema /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1/conf/schema-non-stored-docvalues.xml to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1/conf/schema-non-stored-docvalues.xml.bak
   [junit4]   2> 4880872 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4880877 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4880878 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4880908 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4880931 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test-non-stored-docvalues
   [junit4]   2> 4880938 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test-non-stored-docvalues/1.6 with uniqueid field id
   [junit4]   2> 4880938 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1, trusted=true
   [junit4]   2> 4880938 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4880938 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4880938 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/init-core-data-001/]
   [junit4]   2> 4880941 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4b00cd5a
   [junit4]   2> 4880969 WARN  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, registerPath=/solr,/v2, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4881044 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4881044 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4881046 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=379269927, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 4881046 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7ddaa64a[collection1] main]
   [junit4]   2> 4881047 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1/conf
   [junit4]   2> 4881048 INFO  (coreLoadExecutor-18402-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4881049 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 4881050 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchema Upgraded to managed schema at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1/conf/managed-schema
   [junit4]   2> 4881051 INFO  (searcherExecutor-18403-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7ddaa64a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4881071 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchema Upgraded to managed schema at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-001/collection1/conf/managed-schema
   [junit4]   2> 4881080 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4881080 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:*} 0 0
   [junit4]   2> 4881081 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid1]} 0 0
   [junit4]   2> 4881083 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4881083 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@46757ddb commitCommandVersion:0
   [junit4]   2> 4881149 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b0bdfb9[collection1] main]
   [junit4]   2> 4881150 INFO  (searcherExecutor-18403-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4b0bdfb9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c1)))}
   [junit4]   2> 4881150 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4881150 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 67
   [junit4]   2> 4881152 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:myid*&fl=*&wt=xml} hits=1 status=0 QTime=1
   [junit4]   2> 4881181 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4881181 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:*} 0 28
   [junit4]   2> 4881181 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4881181 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@46757ddb commitCommandVersion:0
   [junit4]   2> 4881207 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@488432f1[collection1] main]
   [junit4]   2> 4881209 INFO  (searcherExecutor-18403-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@488432f1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4881209 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4881210 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 28
   [junit4]   2> 4881210 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 4881210 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1690789497
   [junit4]   2> 4881210 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 4881212 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 4881213 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 4881213 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4881215 INFO  (coreCloseExecutor-18407-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@aaa6c10
   [junit4]   2> 4881216 INFO  (coreCloseExecutor-18407-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=178940944
   [junit4]   2> 4881220 INFO  (TEST-TestUseDocValuesAsStored.testManagedSchema-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testManagedSchema
   [junit4]   2> 4881222 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMultipleSearchResults
   [junit4]   2> 4881225 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 4881254 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4881263 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema-non-stored-docvalues.xml instead
   [junit4]   2> 4881265 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.IndexSchema [null] Schema name=test-non-stored-docvalues
   [junit4]   2> 4881272 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.IndexSchema Loaded schema test-non-stored-docvalues/1.6 with uniqueid field id
   [junit4]   2> 4881273 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchema Upgraded to managed schema at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-002/collection1/conf/managed-schema
   [junit4]   2> 4881276 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema, renamed the non-managed schema /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-002/collection1/conf/schema-non-stored-docvalues.xml to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-002/collection1/conf/schema-non-stored-docvalues.xml.bak
   [junit4]   2> 4881327 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4881338 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4881339 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4881351 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4881360 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test-non-stored-docvalues
   [junit4]   2> 4881367 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test-non-stored-docvalues/1.6 with uniqueid field id
   [junit4]   2> 4881368 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-002/collection1, trusted=true
   [junit4]   2> 4881368 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4881368 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4881368 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/init-core-data-001/]
   [junit4]   2> 4881372 WARN  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, registerPath=/solr,/v2, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4881487 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4881487 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4881489 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6a36a5ba
   [junit4]   2> 4881490 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d5aa130[collection1] main]
   [junit4]   2> 4881500 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-002/collection1/conf
   [junit4]   2> 4881504 INFO  (coreLoadExecutor-18414-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4881506 INFO  (searcherExecutor-18415-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4d5aa130[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4881512 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 4881513 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid1]} 0 0
   [junit4]   2> 4881517 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid2]} 0 3
   [junit4]   2> 4881518 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid3]} 0 0
   [junit4]   2> 4881532 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid4]} 0 14
   [junit4]   2> 4881533 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid5]} 0 0
   [junit4]   2> 4881546 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[myid6]} 0 13
   [junit4]   2> 4881547 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4881547 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@697398b1 commitCommandVersion:0
   [junit4]   2> 4881744 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@1fbc0f3c[collection1] main]
   [junit4]   2> 4881744 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4881746 INFO  (searcherExecutor-18415-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1fbc0f3c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_4(8.0.0):c6)))}
   [junit4]   2> 4881746 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 199
   [junit4]   2> 4881747 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:myid*&fl=*&wt=xml} hits=6 status=0 QTime=0
   [junit4]   2> 4881749 ERROR (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 query failed JSON validation. error=mismatch: 'myid1'!='myid3' @ response/docs/[0]/id
   [junit4]   2>  expected =/response/docs==[{'id':'myid1','test_is_dvo':[101,102,103]},{'id':'myid2','test_is_dvo':[201,202]},{'id':'myid3','test_is_dvo':[301,302,303,304]},{'id':'myid4','test_s_dvo':'hello','test_is_dvo':[401,402]},{'id':'myid5'},{'id':'myid6','test_s_dvo':'hello'}]
   [junit4]   2>  response = {
   [junit4]   2>   "responseHeader":{
   [junit4]   2>     "status":0,
   [junit4]   2>     "QTime":0},
   [junit4]   2>   "response":{"numFound":6,"start":0,"docs":[
   [junit4]   2>       {
   [junit4]   2>         "id":"myid3",
   [junit4]   2>         "test_is_dvo":[301,
   [junit4]   2>           302,
   [junit4]   2>           303,
   [junit4]   2>           304]},
   [junit4]   2>       {
   [junit4]   2>         "id":"myid4",
   [junit4]   2>         "test_s_dvo":"hello",
   [junit4]   2>         "test_is_dvo":[401,
   [junit4]   2>           402]},
   [junit4]   2>       {
   [junit4]   2>         "id":"myid5"},
   [junit4]   2>       {
   [junit4]   2>         "id":"myid6",
   [junit4]   2>         "test_s_dvo":"hello"},
   [junit4]   2>       {
   [junit4]   2>         "id":"myid1",
   [junit4]   2>         "test_is_dvo":[101,
   [junit4]   2>           102,
   [junit4]   2>           103]},
   [junit4]   2>       {
   [junit4]   2>         "id":"myid2",
   [junit4]   2>         "test_is_dvo":[201,
   [junit4]   2>           202]}]
   [junit4]   2>   }}
   [junit4]   2>
   [junit4]   2>  request = q=id:myid*&fl=*&wt=xml
   [junit4]   2> 4881750 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4881750 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:*} 0 0
   [junit4]   2> 4881750 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4881750 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@697398b1 commitCommandVersion:0
   [junit4]   2> 4881831 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@764984ab[collection1] main]
   [junit4]   2> 4881832 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4881833 INFO  (searcherExecutor-18415-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@764984ab[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4881833 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 83
   [junit4]   2> 4881834 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 4881834 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1516764335
   [junit4]   2> 4881834 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 4881836 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 4881863 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 4881864 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4881864 INFO  (coreCloseExecutor-18419-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@36a72b56
   [junit4]   2> 4881865 INFO  (coreCloseExecutor-18419-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=916925270
   [junit4]   2> 4881869 INFO  (TEST-TestUseDocValuesAsStored.testMultipleSearchResults-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMultipleSearchResults
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestUseDocValuesAsStored -Dtests.method=testMultipleSearchResults -Dtests.seed=DD14891C7B505624 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=hi -Dtests.timezone=Indian/Mahe -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.65s J1 | TestUseDocValuesAsStored.testMultipleSearchResults <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: mismatch: 'myid1'!='myid3' @ response/docs/[0]/id
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DD14891C7B505624:EF3E8E8283AE72FD]:0)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:966)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:913)
   [junit4]    > at org.apache.solr.schema.TestUseDocValuesAsStored.testMultipleSearchResults(TestUseDocValuesAsStored.java:243)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 4881880 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRandomSingleAndMultiValued
   [junit4]   2> 4881882 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 4881893 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4881901 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema-non-stored-docvalues.xml instead
   [junit4]   2> 4881903 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.IndexSchema [null] Schema name=test-non-stored-docvalues
   [junit4]   2> 4881928 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.IndexSchema Loaded schema test-non-stored-docvalues/1.6 with uniqueid field id
   [junit4]   2> 4881929 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchema Upgraded to managed schema at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-003/collection1/conf/managed-schema
   [junit4]   2> 4881939 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema, renamed the non-managed schema /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-003/collection1/conf/schema-non-stored-docvalues.xml to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-003/collection1/conf/schema-non-stored-docvalues.xml.bak
   [junit4]   2> 4882004 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4882009 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4882010 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4882022 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 4882031 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test-non-stored-docvalues
   [junit4]   2> 4882058 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test-non-stored-docvalues/1.6 with uniqueid field id
   [junit4]   2> 4882058 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-003/collection1, trusted=true
   [junit4]   2> 4882059 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a9f5185
   [junit4]   2> 4882059 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4882059 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-003/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/init-core-data-001/]
   [junit4]   2> 4882063 WARN  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, registerPath=/solr,/v2, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4882154 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4882155 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4882156 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@34fee8fa
   [junit4]   2> 4882162 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@45ec980d[collection1] main]
   [junit4]   2> 4882163 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001/tempDir-003/collection1/conf
   [junit4]   2> 4882164 INFO  (coreLoadExecutor-18426-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4882165 INFO  (searcherExecutor-18427-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@45ec980d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4882166 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 4882167 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4882167 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:*} 0 0
   [junit4]   2> 4882168 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[1]} 0 0
   [junit4]   2> 4882169 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882169 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882217 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2fe5ff1a[collection1] main]
   [junit4]   2> 4882217 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882218 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2fe5ff1a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(8.0.0):c1)))}
   [junit4]   2> 4882218 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 49
   [junit4]   2> 4882219 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&fl=test_ss_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882227 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&fl=test_ss_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882229 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882235 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882237 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:1&fl=test_ss_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882267 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882269 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[2]} 0 0
   [junit4]   2> 4882269 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882269 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882335 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@f5ffc98[collection1] main]
   [junit4]   2> 4882335 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882337 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@f5ffc98[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(8.0.0):c1) Uninverting(_6(8.0.0):c1)))}
   [junit4]   2> 4882337 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 67
   [junit4]   2> 4882337 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:2&fl=test_i_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882338 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:2&fl=test_i_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882339 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:2&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882339 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:2&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882340 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:2&fl=test_i_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882340 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=2 status=0 QTime=0
   [junit4]   2> 4882342 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[3]} 0 0
   [junit4]   2> 4882342 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882342 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882416 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7ffb08e9[collection1] main]
   [junit4]   2> 4882416 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882420 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7ffb08e9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3)))}
   [junit4]   2> 4882433 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 91
   [junit4]   2> 4882434 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:3&fl=test_d_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882443 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:3&fl=test_d_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882444 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:3&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882445 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:3&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882445 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:3&fl=test_d_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882446 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=3 status=0 QTime=0
   [junit4]   2> 4882447 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[4]} 0 0
   [junit4]   2> 4882448 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882448 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882528 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@43c6c9c8[collection1] main]
   [junit4]   2> 4882528 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882529 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@43c6c9c8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3) Uninverting(_9(8.0.0):c1)))}
   [junit4]   2> 4882530 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 82
   [junit4]   2> 4882530 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:4&fl=test_l_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882531 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:4&fl=test_l_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882532 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:4&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882532 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:4&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882533 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:4&fl=test_l_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882533 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=4 status=0 QTime=0
   [junit4]   2> 4882535 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5]} 0 1
   [junit4]   2> 4882536 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882536 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882569 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@76915466[collection1] main]
   [junit4]   2> 4882570 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882571 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@76915466[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3) Uninverting(_9(8.0.0):c1) Uninverting(_a(8.0.0):c1)))}
   [junit4]   2> 4882571 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 35
   [junit4]   2> 4882572 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:5&fl=test_f_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882572 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:5&fl=test_f_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882589 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:5&fl=*dvo&wt=xml} hits=1 status=0 QTime=9
   [junit4]   2> 4882590 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:5&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882590 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:5&fl=test_f_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882591 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=5 status=0 QTime=0
   [junit4]   2> 4882593 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[6]} 0 0
   [junit4]   2> 4882593 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882593 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882623 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@50d3c3ce[collection1] main]
   [junit4]   2> 4882625 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@50d3c3ce[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3) Uninverting(_9(8.0.0):c1) Uninverting(_a(8.0.0):c1) Uninverting(_b(8.0.0):c1)))}
   [junit4]   2> 4882625 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882625 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 32
   [junit4]   2> 4882626 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:6&fl=test_dts_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882627 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:6&fl=test_dts_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882628 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:6&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882653 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:6&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882654 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:6&fl=test_dts_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882655 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=6 status=0 QTime=0
   [junit4]   2> 4882657 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[7]} 0 0
   [junit4]   2> 4882657 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882657 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882707 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@172196f8[collection1] main]
   [junit4]   2> 4882708 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882709 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@172196f8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3) Uninverting(_9(8.0.0):c1) Uninverting(_a(8.0.0):c1) Uninverting(_b(8.0.0):c1) Uninverting(_c(8.0.0):c1)))}
   [junit4]   2> 4882709 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 52
   [junit4]   2> 4882710 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:7&fl=test_s_dv&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882710 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:7&fl=test_s_dv,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882711 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:7&fl=*_dv&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882712 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:7&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882712 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:7&fl=test_s_dv,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882713 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=7 status=0 QTime=0
   [junit4]   2> 4882714 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[8]} 0 0
   [junit4]   2> 4882714 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882714 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882762 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@58266762[collection1] main]
   [junit4]   2> 4882762 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882763 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@58266762[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3) Uninverting(_9(8.0.0):c1) Uninverting(_a(8.0.0):c1) Uninverting(_b(8.0.0):c1) Uninverting(_c(8.0.0):c1) Uninverting(_d(8.0.0):c1)))}
   [junit4]   2> 4882763 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 49
   [junit4]   2> 4882764 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:8&fl=test_s_dvo2&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882765 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:8&fl=test_s_dvo2,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882765 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:8&fl=*vo2&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882766 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:8&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882767 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:8&fl=test_s_dvo2,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882767 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=8 status=0 QTime=0
   [junit4]   2> 4882770 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[9]} 0 0
   [junit4]   2> 4882770 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882770 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882819 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d4b366a[collection1] main]
   [junit4]   2> 4882820 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882821 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4d4b366a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8(8.0.0):c3) Uninverting(_9(8.0.0):c1) Uninverting(_a(8.0.0):c1) Uninverting(_b(8.0.0):c1) Uninverting(_c(8.0.0):c1) Uninverting(_d(8.0.0):c1) Uninverting(_e(8.0.0):c1)))}
   [junit4]   2> 4882821 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 51
   [junit4]   2> 4882822 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:9&fl=enums_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882824 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:9&fl=enums_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882826 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:9&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882836 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:9&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882838 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:9&fl=enums_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882850 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=9 status=0 QTime=0
   [junit4]   2> 4882852 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4882852 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:*} 0 0
   [junit4]   2> 4882853 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[10]} 0 0
   [junit4]   2> 4882854 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882854 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882918 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@c6579fb[collection1] main]
   [junit4]   2> 4882918 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882920 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@c6579fb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_f(8.0.0):c1)))}
   [junit4]   2> 4882924 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 70
   [junit4]   2> 4882925 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&fl=test_s_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882925 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&fl=test_s_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882926 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882926 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882927 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:10&fl=test_s_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882927 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882928 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[11]} 0 0
   [junit4]   2> 4882929 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882929 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4882969 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2413717d[collection1] main]
   [junit4]   2> 4882970 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4882972 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2413717d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_h(8.0.0):c2)))}
   [junit4]   2> 4882975 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 46
   [junit4]   2> 4882976 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:11&fl=test_i_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882976 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:11&fl=test_i_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882977 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:11&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882977 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:11&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882978 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:11&fl=test_i_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4882978 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=2 status=0 QTime=0
   [junit4]   2> 4882980 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[12]} 0 0
   [junit4]   2> 4882980 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4882980 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4883141 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@36b17b0c[collection1] main]
   [junit4]   2> 4883141 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4883142 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@36b17b0c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_h(8.0.0):c2) Uninverting(_i(8.0.0):c1)))}
   [junit4]   2> 4883147 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 167
   [junit4]   2> 4883148 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:12&fl=test_ds_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883149 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:12&fl=test_ds_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883150 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:12&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883154 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:12&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883163 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:12&fl=test_ds_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883164 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=3 status=0 QTime=0
   [junit4]   2> 4883166 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[13]} 0 1
   [junit4]   2> 4883167 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4883167 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4883190 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@69be5216[collection1] main]
   [junit4]   2> 4883191 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4883192 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@69be5216[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_h(8.0.0):c2) Uninverting(_i(8.0.0):c1) Uninverting(_j(8.0.0):c1)))}
   [junit4]   2> 4883192 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 25
   [junit4]   2> 4883193 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:13&fl=test_l_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883193 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:13&fl=test_l_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883194 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:13&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883194 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:13&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883195 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:13&fl=test_l_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883196 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=4 status=0 QTime=0
   [junit4]   2> 4883197 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[14]} 0 0
   [junit4]   2> 4883198 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4883198 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4883247 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c48d9bf[collection1] main]
   [junit4]   2> 4883248 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4883249 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c48d9bf[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_h(8.0.0):c2) Uninverting(_i(8.0.0):c1) Uninverting(_j(8.0.0):c1) Uninverting(_k(8.0.0):c1)))}
   [junit4]   2> 4883249 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 51
   [junit4]   2> 4883249 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&fl=test_fs_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883251 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&fl=test_fs_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883253 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883254 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883255 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:14&fl=test_fs_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883261 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=5 status=0 QTime=0
   [junit4]   2> 4883263 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[15]} 0 0
   [junit4]   2> 4883264 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4883264 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4883330 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@28df9dfe[collection1] main]
   [junit4]   2> 4883331 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 4883332 INFO  (searcherExecutor-18427-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@28df9dfe[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_h(8.0.0):c2) Uninverting(_i(8.0.0):c1) Uninverting(_j(8.0.0):c1) Uninverting(_k(8.0.0):c1) Uninverting(_l(8.0.0):c1)))}
   [junit4]   2> 4883339 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 75
   [junit4]   2> 4883340 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:15&fl=test_dts_dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883341 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:15&fl=test_dts_dvo,*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883350 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:15&fl=*dvo&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883351 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:15&fl=*&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883353 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:15&fl=test_dts_dvo,fakeFieldName&wt=xml} hits=1 status=0 QTime=0
   [junit4]   2> 4883354 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=*:*&fl=*&wt=xml} hits=6 status=0 QTime=0
   [junit4]   2> 4883357 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[16]} 0 0
   [junit4]   2> 4883357 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4883357 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6d267cab commitCommandVersion:0
   [junit4]   2> 4883408 INFO  (TEST-TestUseDocValuesAsStored.testRandomSingleAndMultiValued-seed#[DD14891C7B505624]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ca5c23

[...truncated too long message...]

o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 4897199 INFO  (TEST-TestUseDocValuesAsStored.testMultivaluedOrdering-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 4897200 INFO  (TEST-TestUseDocValuesAsStored.testMultivaluedOrdering-seed#[DD14891C7B505624]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 4897201 INFO  (TEST-TestUseDocValuesAsStored.testMultivaluedOrdering-seed#[DD14891C7B505624]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 4897202 INFO  (coreCloseExecutor-18467-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@68119c92
   [junit4]   2> 4897204 INFO  (coreCloseExecutor-18467-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1745984658
   [junit4]   2> 4897213 INFO  (TEST-TestUseDocValuesAsStored.testMultivaluedOrdering-seed#[DD14891C7B505624]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMultivaluedOrdering
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.schema.TestUseDocValuesAsStored_DD14891C7B505624-001
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=32661, maxDocsPerChunk=203, blockSize=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=32661, blockSize=7)), sim=RandomSimilarity(queryNorm=true): {}, locale=hi, timezone=Indian/Mahe
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=193367600,total=532152320
   [junit4]   2> NOTE: All tests run in this JVM: [LeaderInitiatedRecoveryOnCommitTest, DisMaxRequestHandlerTest, TestConfigSetProperties, DOMUtilTest, TestManagedSynonymFilterFactory, TestNoOpRegenerator, TestTestInjection, OverseerCollectionConfigSetProcessorTest, TestReloadAndDeleteDocs, DirectSolrConnectionTest, DistributedFacetPivotLongTailTest, GraphQueryTest, TestCloudJSONFacetJoinDomain, CloudMLTQParserTest, TestClusterStateMutator, TestDynamicFieldResource, BasicAuthIntegrationTest, FileBasedSpellCheckerTest, HdfsBasicDistributedZk2Test, TestSerializedLuceneMatchVersion, ImplicitSnitchTest, TestFieldCacheSort, TestLRUCache, TestNumericTokenStream, TestFiltering, LeaderElectionContextKeyTest, IndexSchemaRuntimeFieldTest, TestCryptoKeys, TestAuthorizationFramework, SchemaVersionSpecificBehaviorTest, SolrMetricReporterTest, TestComplexPhraseLeadingWildcard, RecoveryAfterSoftCommitTest, TestBinaryResponseWriter, TestSSLRandomization, TestUnifiedSolrHighlighter, TestHalfAndHalfDocValues, BufferStoreTest, CollectionsAPIAsyncDistributedZkTest, TestCoreContainer, TestDownShardTolerantSearch, TestDFISimilarityFactory, TestTrieFacet, TestPartialUpdateDeduplication, TestManagedResourceStorage, SolrGraphiteReporterTest, TestBM25SimilarityFactory, DeleteStatusTest, TestJsonFacetRefinement, TestTolerantUpdateProcessorRandomCloud, ResourceLoaderTest, SpellCheckComponentTest, TestExactSharedStatsCache, TestBackupRepositoryFactory, TestQueryTypes, TestSweetSpotSimilarityFactory, SolrCLIZkUtilsTest, TestScoreJoinQPNoScore, ExitableDirectoryReaderTest, TermVectorComponentTest, DocValuesTest, BooleanFieldTest, BigEndianAscendingWordDeserializerTest, TestSolrIndexConfig, TestSolrJ, TestSha256AuthenticationProvider, ExplicitHLLTest, TestHighFrequencyDictionaryFactory, CollectionReloadTest, AsyncCallRequestStatusResponseTest, DocValuesNotIndexedTest, SOLR749Test, TestConfigSetsAPI, LoggingHandlerTest, TestMacros, TestReloadDeadlock, TestFieldSortValues, BlobRepositoryCloudTest, CustomCollectionTest, TestCursorMarkWithoutUniqueKey, TestFastLRUCache, SortSpecParsingTest, ScriptEngineTest, TestCustomStream, TestOrdValues, TestSimpleTrackingShardHandler, TestSQLHandler, HdfsRecoverLeaseTest, TestInPlaceUpdatesStandalone, ConjunctionSolrSpellCheckerTest, LargeFieldTest, ClusterStateUpdateTest, TestSchemalessBufferedUpdates, StatelessScriptUpdateProcessorFactoryTest, DistributedExpandComponentTest, TestMaxScoreQueryParser, TestPrepRecovery, SuggestComponentContextFilterQueryTest, ZkSolrClientTest, TestMultiValuedNumericRangeQuery, WrapperMergePolicyFactoryTest, SolrCmdDistributorTest, DocValuesMultiTest, TestReplicaProperties, TestBlobHandler, SharedFSAutoReplicaFailoverTest, TestSolrConfigHandlerConcurrent, CacheHeaderTest, TestUpdate, DistributedSuggestComponentTest, BlockJoinFacetDistribTest, TestFieldTypeCollectionResource, TestDocTermOrds, TestCloudRecovery, TestExceedMaxTermLength, MissingSegmentRecoveryTest, TestAuthenticationFramework, MetricsConfigTest, TestSchemaResource, DistribJoinFromCollectionTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, AutoCommitTest, TestDistributedGrouping, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestJoin, TestStressVersions, DistributedTermsComponentTest, TestRangeQuery, StatsComponentTest, TestGroupingSearch, PeerSyncTest, ConvertedLegacyTest, TestSort, BasicFunctionalityTest, HighlighterTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SuggesterFSTTest, CoreAdminHandlerTest, SuggesterTSTTest, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, TestRemoteStreaming, DistanceFunctionTest, TestSolrDeletionPolicy1, DebugComponentTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, RequestHandlersTest, TestValueSourceCache, TestIndexingPerformance, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, SolrPluginUtilsTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, QueryParsingTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, CSVRequestHandlerTest, SearchHandlerTest, TestLFUCache, HighlighterConfigTest, TestQuerySenderListener, UpdateParamsTest, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, BadComponentTest, TestStressRecovery, TestMergePolicyConfig, SampleTest, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestCharFilters, TestXIncludeConfig, EchoParamsTest, TestPerFieldSimilarity, TestIBSimilarityFactory, TimeZoneUtilsTest, TestFastOutputStream, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, DateFieldTest, SpellingQueryConverterTest, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, FileUtilsTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, CursorPagingTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, ActionThrottleTest, AliasIntegrationTest, AssignTest, BaseCdcrDistributedZkTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CdcrVersionReplicationTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, ConcurrentDeleteAndCreateCollectionTest, ForceLeaderTest, SolrCloudExampleTest, TestCloudDeleteByQuery, TestCloudPivotFacet, TestConfigSetsAPIExclusivity, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestExclusionRuleCollectionAccess, TestHdfsCloudBackupRestore, TestLeaderElectionWithEmptyReplica, TestLeaderInitiatedRecoveryThread, TestLocalFSCloudBackupRestore, TestLockTree, TestMiniSolrCloudCluster, TestPullReplica, TestPullReplicaErrorHandling, TestRequestForwarding, TestSegmentSorting, TestSizeLimitedDistributedMap, TestStressLiveNodes, TlogReplayBufferedWhileIndexingTest, HdfsWriteToMultipleCollectionsTest, ZkStateWriterTest, RuleEngineTest, ConfigureRecoveryStrategyTest, DirectoryFactoryTest, HdfsDirectoryFactoryTest, QueryResultKeyTest, TestDynamicLoading, TestSolrCoreSnapshots, TestRestoreCore, V2StandaloneTest, CoreAdminCreateDiscoverTest, CoreAdminRequestStatusTest, CoreMergeIndexesAdminHandlerTest, MetricsHandlerTest, SegmentsInfoRequestHandlerTest, TestApiFramework, TestCollectionAPIs, TestConfigsApi, DistributedFacetPivotSmallTest, DistributedFacetPivotWhiteBoxTest, InfixSuggestersTest, TestExpandComponent, TestHttpShardHandlerFactory, TestTrackingShardHandlerFactory, SolrCloudReportersTest, TestMacroExpander, JSONWriterTest, SmileWriterTest, TestExportWriter, TestSolrQueryResponse, TestSubQueryTransformerCrossCore, TestRestManager, TestClassNameShortening, TestDynamicFieldCollectionResource, TestFieldCollectionResource, TestManagedStopFilterFactory, TestManagedSynonymGraphFilterFactory, BadCopyFieldTest, DateRangeFieldTest, DocValuesMissingTest, EnumFieldTest, ManagedSchemaRoundRobinCloudTest, TestManagedSchemaAPI, TestManagedSchemaThreadSafety, TestSchemaManager, TestUseDocValuesAsStored]
   [junit4] Completed [651/725 (1!)] on J1 in 16.54s, 6 tests, 1 error <<< FAILURES!

[...truncated 43448 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1346 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1346/

2 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([816B775345BBAEC2:93F4889EB47C33A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([816B775345BBAEC2:93F4889EB47C33A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11696 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/init-core-data-001
   [junit4]   2> 564709 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 564709 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 564711 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 564712 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 564746 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 564752 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 564753 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 564768 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_38599_hdfs____.p89j2e/webapp
   [junit4]   2> 565143 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38599
   [junit4]   2> 565202 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 565204 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 565217 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56576_datanode____lvpat6/webapp
   [junit4]   2> 565620 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56576
   [junit4]   2> 565654 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 565656 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 565669 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51669_datanode____bsvumc/webapp
   [junit4]   2> 565722 INFO  (IPC Server handler 3 on 59001) [    ] BlockStateChange BLOCK* processReport: from storage DS-a49a44d9-3411-4aee-97f9-bbdda2c95e08 node DatanodeRegistration(127.0.0.1:58939, datanodeUuid=378ce1a4-60a5-4187-9bf2-63cb1a15eea2, infoPort=39045, infoSecurePort=0, ipcPort=36319, storageInfo=lv=-56;cid=testClusterID;nsid=1105079723;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 565722 INFO  (IPC Server handler 3 on 59001) [    ] BlockStateChange BLOCK* processReport: from storage DS-8b2c8039-2b1a-4cea-8cbe-ca2418e058c3 node DatanodeRegistration(127.0.0.1:58939, datanodeUuid=378ce1a4-60a5-4187-9bf2-63cb1a15eea2, infoPort=39045, infoSecurePort=0, ipcPort=36319, storageInfo=lv=-56;cid=testClusterID;nsid=1105079723;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 566067 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[816B775345BBAEC2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51669
   [junit4]   2> 566164 INFO  (IPC Server handler 8 on 59001) [    ] BlockStateChange BLOCK* processReport: from storage DS-a6058b5d-2366-4377-8b07-b7076ab0ab24 node DatanodeRegistration(127.0.0.1:41078, datanodeUuid=821402ea-3062-4acf-91fc-4b31099b5b85, infoPort=45121, infoSecurePort=0, ipcPort=37002, storageInfo=lv=-56;cid=testClusterID;nsid=1105079723;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 566164 INFO  (IPC Server handler 8 on 59001) [    ] BlockStateChange BLOCK* processReport: from storage DS-21d26fd5-9195-489b-bae0-d7065dc0ed5d node DatanodeRegistration(127.0.0.1:41078, datanodeUuid=821402ea-3062-4acf-91fc-4b31099b5b85, infoPort=45121, infoSecurePort=0, ipcPort=37002, storageInfo=lv=-56;cid=testClusterID;nsid=1105079723;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 566250 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 566250 INFO  (Thread-14201) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 566250 INFO  (Thread-14201) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 566254 ERROR (Thread-14201) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 566350 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkTestServer start zk server on port:46882
   [junit4]   2> 566366 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 566368 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 566370 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 566371 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 566373 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 566374 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 566376 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 566377 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 566378 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 566380 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 566381 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 566386 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 566544 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 566546 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64089399{/,null,AVAILABLE}
   [junit4]   2> 566546 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@32c108da{SSL,[ssl, http/1.1]}{127.0.0.1:52488}
   [junit4]   2> 566547 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server Started @569860ms
   [junit4]   2> 566547 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:59001/hdfs__localhost_59001__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001_tempDir-002_control_data, replicaType=NRT, hostContext=/, hostPort=52488, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/control-001/cores}
   [junit4]   2> 566547 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 566548 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 566548 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 566548 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 566548 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T10:59:59.887Z
   [junit4]   2> 566550 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 566551 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/control-001/solr.xml
   [junit4]   2> 566567 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 566571 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46882/solr
   [junit4]   2> 566681 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 566681 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52488_
   [junit4]   2> 566682 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.c.Overseer Overseer (id=98255005259530245-127.0.0.1:52488_-n_0000000000) starting
   [junit4]   2> 566697 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52488_
   [junit4]   2> 566700 INFO  (zkCallback-890-thread-1-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 566828 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 566834 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 566834 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 566836 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:52488_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/control-001/cores
   [junit4]   2> 566879 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 566880 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46882/solr ready
   [junit4]   2> 566917 INFO  (qtp2087241229-18341) [n:127.0.0.1:52488_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:52488_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 566921 INFO  (OverseerThreadFactory-2037-thread-1-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 567032 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 567032 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 567137 INFO  (zkCallback-890-thread-1-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 568052 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 568069 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 568199 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 568217 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 568218 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 568220 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:59001/solr_hdfs_home
   [junit4]   2> 568220 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 568220 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 568220 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:59001/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 568235 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 568247 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 568247 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 568247 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 568275 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 568277 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 568295 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 568302 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 568302 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 568302 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 568317 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 568318 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5940b760
   [junit4]   2> 568378 INFO  (IPC Server handler 2 on 59001) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41078 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a49a44d9-3411-4aee-97f9-bbdda2c95e08:NORMAL:127.0.0.1:58939|RBW], ReplicaUC[[DISK]DS-a6058b5d-2366-4377-8b07-b7076ab0ab24:NORMAL:127.0.0.1:41078|RBW]]} size 0
   [junit4]   2> 568378 INFO  (IPC Server handler 3 on 59001) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58939 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a6058b5d-2366-4377-8b07-b7076ab0ab24:NORMAL:127.0.0.1:41078|RBW], ReplicaUC[[DISK]DS-8b2c8039-2b1a-4cea-8cbe-ca2418e058c3:NORMAL:127.0.0.1:58939|FINALIZED]]} size 0
   [junit4]   2> 568393 WARN  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 568469 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 568469 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 568469 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 568480 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 568480 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 568482 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=0.8271484375, floorSegmentMB=1.505859375, forceMergeDeletesPctAllowed=7.251543243114843, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 568489 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c4d4e3f[control_collection_shard1_replica_n1] main]
   [junit4]   2> 568491 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 568491 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 568492 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 568494 INFO  (searcherExecutor-2040-thread-1-processing-n:127.0.0.1:52488_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@4c4d4e3f[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 568494 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572080086504439808
   [junit4]   2> 568501 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 568501 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 568502 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:52488/control_collection_shard1_replica_n1/
   [junit4]   2> 568502 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 568502 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:52488/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 568502 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 568506 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:52488/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 568609 INFO  (zkCallback-890-thread-1-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 568657 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 568661 INFO  (qtp2087241229-18347) [n:127.0.0.1:52488_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1629
   [junit4]   2> 568674 INFO  (qtp2087241229-18341) [n:127.0.0.1:52488_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 568776 INFO  (zkCallback-890-thread-1-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 568922 INFO  (OverseerCollectionConfigSetProcessor-98255005259530245-127.0.0.1:52488_-n_0000000000) [n:127.0.0.1:52488_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 569674 INFO  (qtp2087241229-18341) [n:127.0.0.1:52488_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:52488_&wt=javabin&version=2} status=0 QTime=2757
   [junit4]   2> 569692 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 569693 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46882/solr ready
   [junit4]   2> 569693 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 569696 INFO  (qtp2087241229-18344) [n:127.0.0.1:52488_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=4&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 569707 INFO  (OverseerThreadFactory-2037-thread-2-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 569708 WARN  (OverseerThreadFactory-2037-thread-2-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 569919 INFO  (qtp2087241229-18344) [n:127.0.0.1:52488_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 569919 INFO  (qtp2087241229-18344) [n:127.0.0.1:52488_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=4&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=222
   [junit4]   2> 570079 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-1-001 of type NRT
   [junit4]   2> 570085 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 570098 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55f4fc7d{/,null,AVAILABLE}
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@feb4bf3{SSL,[ssl, http/1.1]}{127.0.0.1:32976}
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server Started @573412ms
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:59001/hdfs__localhost_59001__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001_tempDir-002_jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=32976, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-1-001/cores}
   [junit4]   2> 570099 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 570099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T11:00:03.438Z
   [junit4]   2> 570111 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 570111 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-1-001/solr.xml
   [junit4]   2> 570118 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 570123 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46882/solr
   [junit4]   2> 570187 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 570189 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 570191 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32976_
   [junit4]   2> 570193 INFO  (zkCallback-890-thread-1-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 570215 INFO  (zkCallback-897-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 570215 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 570299 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 570308 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 570308 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 570310 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:32976_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-1-001/cores
   [junit4]   2> 570386 INFO  (qtp210925036-18398) [n:127.0.0.1:32976_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:32976_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 570396 INFO  (OverseerCollectionConfigSetProcessor-98255005259530245-127.0.0.1:52488_-n_0000000000) [n:127.0.0.1:52488_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 570403 INFO  (OverseerThreadFactory-2037-thread-3-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:32976_ for creating new replica
   [junit4]   2> 570408 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 570409 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 570515 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 571428 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 571447 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema [collection1_shard2_replica_n0] Schema name=test
   [junit4]   2> 571614 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 571634 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 571635 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n0' (registry 'solr.core.collection1.shard2.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 571635 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:59001/solr_hdfs_home
   [junit4]   2> 571635 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 571635 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 571635 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [[collection1_shard2_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-1-001/cores/collection1_shard2_replica_n0], dataDir=[hdfs://localhost:59001/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 571637 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 571646 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 571646 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 571646 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 571654 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 571655 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 571675 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 571682 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 571682 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 571682 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 571694 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 571694 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4383d4b6
   [junit4]   2> 571716 INFO  (IPC Server handler 3 on 59001) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41078 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-8b2c8039-2b1a-4cea-8cbe-ca2418e058c3:NORMAL:127.0.0.1:58939|RBW], ReplicaUC[[DISK]DS-21d26fd5-9195-489b-bae0-d7065dc0ed5d:NORMAL:127.0.0.1:41078|FINALIZED]]} size 0
   [junit4]   2> 571717 INFO  (IPC Server handler 4 on 59001) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58939 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-21d26fd5-9195-489b-bae0-d7065dc0ed5d:NORMAL:127.0.0.1:41078|FINALIZED], ReplicaUC[[DISK]DS-a49a44d9-3411-4aee-97f9-bbdda2c95e08:NORMAL:127.0.0.1:58939|FINALIZED]]} size 0
   [junit4]   2> 571732 WARN  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 571797 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 571797 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 571797 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 571808 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 571808 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 571811 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=0.8271484375, floorSegmentMB=1.505859375, forceMergeDeletesPctAllowed=7.251543243114843, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 571822 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@28ed3358[collection1_shard2_replica_n0] main]
   [junit4]   2> 571823 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 571824 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 571825 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 571838 INFO  (searcherExecutor-2051-thread-1-processing-n:127.0.0.1:32976_ x:collection1_shard2_replica_n0 s:shard2 c:collection1) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SolrCore [collection1_shard2_replica_n0] Registered new searcher Searcher@28ed3358[collection1_shard2_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 571838 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572080090010877952
   [junit4]   2> 571847 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 571847 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 571847 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:32976/collection1_shard2_replica_n0/
   [junit4]   2> 571847 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 571847 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:32976/collection1_shard2_replica_n0/ has no replicas
   [junit4]   2> 571847 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 571852 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:32976/collection1_shard2_replica_n0/ shard2
   [junit4]   2> 571953 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 572003 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 572005 INFO  (qtp210925036-18395) [n:127.0.0.1:32976_ c:collection1 s:shard2  x:collection1_shard2_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n0&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1596
   [junit4]   2> 572008 INFO  (qtp210925036-18398) [n:127.0.0.1:32976_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:32976_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1622
   [junit4]   2> 572108 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 572163 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-2-001 of type NRT
   [junit4]   2> 572164 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 572166 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@608a7ffd{/,null,AVAILABLE}
   [junit4]   2> 572166 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@922b906{SSL,[ssl, http/1.1]}{127.0.0.1:48769}
   [junit4]   2> 572166 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server Started @575480ms
   [junit4]   2> 572167 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:59001/hdfs__localhost_59001__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001_tempDir-002_jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=48769, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-2-001/cores}
   [junit4]   2> 572170 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 572171 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 572171 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 572171 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 572171 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T11:00:05.510Z
   [junit4]   2> 572174 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 572174 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-2-001/solr.xml
   [junit4]   2> 572182 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 572185 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46882/solr
   [junit4]   2> 572195 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 572198 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 572201 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48769_
   [junit4]   2> 572202 INFO  (zkCallback-890-thread-2-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 572202 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 572202 INFO  (zkCallback-897-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 572204 INFO  (zkCallback-908-thread-1-processing-n:127.0.0.1:48769_) [n:127.0.0.1:48769_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 572266 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 572273 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 572273 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 572274 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:48769_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-2-001/cores
   [junit4]   2> 572307 INFO  (qtp210925036-18399) [n:127.0.0.1:32976_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:48769_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 572310 INFO  (OverseerCollectionConfigSetProcessor-98255005259530245-127.0.0.1:52488_-n_0000000000) [n:127.0.0.1:52488_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 572310 INFO  (OverseerThreadFactory-2037-thread-4-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:48769_ for creating new replica
   [junit4]   2> 572342 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n0&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 572343 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 572447 INFO  (zkCallback-908-thread-1-processing-n:127.0.0.1:48769_) [n:127.0.0.1:48769_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 572447 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 573360 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 573375 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.IndexSchema [collection1_shard3_replica_n0] Schema name=test
   [junit4]   2> 573488 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 573504 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_n0' using configuration from collection collection1, trusted=true
   [junit4]   2> 573504 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_n0' (registry 'solr.core.collection1.shard3.replica_n0') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 573504 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:59001/solr_hdfs_home
   [junit4]   2> 573505 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 573505 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 573505 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrCore [[collection1_shard3_replica_n0] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-2-001/cores/collection1_shard3_replica_n0], dataDir=[hdfs://localhost:59001/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 573506 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 573520 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 573520 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 573520 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 573532 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 573534 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 573556 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59001/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 573563 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 573563 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 573563 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 573570 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 573571 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@13efbbf3
   [junit4]   2> 573583 INFO  (IPC Server handler 6 on 59001) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58939 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-21d26fd5-9195-489b-bae0-d7065dc0ed5d:NORMAL:127.0.0.1:41078|RBW], ReplicaUC[[DISK]DS-8b2c8039-2b1a-4cea-8cbe-ca2418e058c3:NORMAL:127.0.0.1:58939|FINALIZED]]} size 0
   [junit4]   2> 573583 INFO  (IPC Server handler 7 on 59001) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41078 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-8b2c8039-2b1a-4cea-8cbe-ca2418e058c3:NORMAL:127.0.0.1:58939|FINALIZED], ReplicaUC[[DISK]DS-a6058b5d-2366-4377-8b07-b7076ab0ab24:NORMAL:127.0.0.1:41078|FINALIZED]]} size 0
   [junit4]   2> 573589 WARN  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 573646 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 573646 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 573646 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 573657 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 573657 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 573659 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=0.8271484375, floorSegmentMB=1.505859375, forceMergeDeletesPctAllowed=7.251543243114843, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 573666 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.SolrIndexSearcher Opening [Searcher@3de0c843[collection1_shard3_replica_n0] main]
   [junit4]   2> 573667 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 573668 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 573668 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 573670 INFO  (searcherExecutor-2062-thread-1-processing-n:127.0.0.1:48769_ x:collection1_shard3_replica_n0 s:shard3 c:collection1) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SolrCore [collection1_shard3_replica_n0] Registered new searcher Searcher@3de0c843[collection1_shard3_replica_n0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 573670 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572080091931869184
   [junit4]   2> 573677 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 573677 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 573677 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:48769/collection1_shard3_replica_n0/
   [junit4]   2> 573678 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 573678 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.SyncStrategy https://127.0.0.1:48769/collection1_shard3_replica_n0/ has no replicas
   [junit4]   2> 573678 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 573683 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:48769/collection1_shard3_replica_n0/ shard3
   [junit4]   2> 573786 INFO  (zkCallback-908-thread-1-processing-n:127.0.0.1:48769_) [n:127.0.0.1:48769_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 573786 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 573834 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 573837 INFO  (qtp191489558-18437) [n:127.0.0.1:48769_ c:collection1 s:shard3  x:collection1_shard3_replica_n0] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n0&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1494
   [junit4]   2> 573840 INFO  (qtp210925036-18399) [n:127.0.0.1:32976_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:48769_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2} status=0 QTime=1532
   [junit4]   2> 573939 INFO  (zkCallback-908-thread-1-processing-n:127.0.0.1:48769_) [n:127.0.0.1:48769_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 573939 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 573989 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-3-001 of type NRT
   [junit4]   2> 573993 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 573995 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@33edac9f{/,null,AVAILABLE}
   [junit4]   2> 573995 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@150221ef{SSL,[ssl, http/1.1]}{127.0.0.1:54069}
   [junit4]   2> 573995 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.Server Started @577309ms
   [junit4]   2> 573997 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:59001/hdfs__localhost_59001__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001_tempDir-002_jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=54069, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-3-001/cores}
   [junit4]   2> 573997 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 573998 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 573998 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 573998 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 573998 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-05T11:00:07.337Z
   [junit4]   2> 574001 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 574001 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-3-001/solr.xml
   [junit4]   2> 574007 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 574010 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46882/solr
   [junit4]   2> 574023 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 574025 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 574027 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54069_
   [junit4]   2> 574029 INFO  (zkCallback-897-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 574029 INFO  (zkCallback-890-thread-2-processing-n:127.0.0.1:52488_) [n:127.0.0.1:52488_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 574030 INFO  (zkCallback-908-thread-1-processing-n:127.0.0.1:48769_) [n:127.0.0.1:48769_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 574030 INFO  (zkCallback-902-thread-2-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 574031 INFO  (zkCallback-914-thread-1-processing-n:127.0.0.1:54069_) [n:127.0.0.1:54069_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 574170 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 574176 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 574176 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@55a1ff62
   [junit4]   2> 574178 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [n:127.0.0.1:54069_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_816B775345BBAEC2-001/shard-3-001/cores
   [junit4]   2> 574219 INFO  (qtp210925036-18397) [n:127.0.0.1:32976_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:54069_&action=ADDREPLICA&collection=collection1&shard=shard4&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 574222 INFO  (OverseerCollectionConfigSetProcessor-98255005259530245-127.0.0.1:52488_-n_0000000000) [n:127.0.0.1:52488_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-00000000

[...truncated too long message...]

d#[816B775345BBAEC2]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6291b07a{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1927876 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44c7d1f2{/,null,UNAVAILABLE}
   [junit4]   2> 1927877 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ChaosMonkey monkey: stop jetty! 44482
   [junit4]   2> 1927877 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=50704302
   [junit4]   2> 1927877 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1927877 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1927878 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1927879 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1927880 INFO  (coreCloseExecutor-4743-thread-1) [n:127.0.0.1:44482_ c:collection1 s:shard2 r:core_node13 x:collection1_shard2_replica_n3] o.a.s.c.SolrCore [collection1_shard2_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@72a2b0b8
   [junit4]   2> 1927882 INFO  (coreCloseExecutor-4743-thread-1) [n:127.0.0.1:44482_ c:collection1 s:shard2 r:core_node13 x:collection1_shard2_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_n3, tag=1923264696
   [junit4]   2> 1927886 INFO  (coreCloseExecutor-4743-thread-1) [n:127.0.0.1:44482_ c:collection1 s:shard2 r:core_node13 x:collection1_shard2_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=1923264696
   [junit4]   2> 1927887 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.Overseer Overseer (id=98255089138466862-127.0.0.1:44482_-n_0000000013) closing
   [junit4]   2> 1927887 INFO  (OverseerStateUpdate-98255089138466862-127.0.0.1:44482_-n_0000000013) [n:127.0.0.1:44482_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:44482_
   [junit4]   2> 1929388 WARN  (zkCallback-2331-thread-2-processing-n:127.0.0.1:44482_) [n:127.0.0.1:44482_    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1929389 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@fba1cce{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1929389 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1225fcd6{/,null,UNAVAILABLE}
   [junit4]   2> 1929390 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1929392 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[816B775345BBAEC2]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55167 55167
   [junit4]   2> 1929558 INFO  (Thread-18539) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55167 55167
   [junit4]   2> 1929559 WARN  (Thread-18539) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 16 /solr/aliases.json
   [junit4]   2> 14 /solr/security.json
   [junit4]   2> 14 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 16 /solr/clusterprops.json
   [junit4]   2> 16 /solr/clusterstate.json
   [junit4]   2> 13 /solr/collections/collection1/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 16 /solr/live_nodes
   [junit4]   2> 16 /solr/collections
   [junit4]   2> 12 /solr/overseer/collection-queue-work
   [junit4]   2> 9 /solr/overseer/queue
   [junit4]   2> 4 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=test -Dtests.seed=816B775345BBAEC2 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=en -Dtests.timezone=Africa/Ceuta -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 83.4s J2 | ChaosMonkeySafeLeaderTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
   [junit4]    > at __randomizedtesting.SeedInfo.seed([816B775345BBAEC2:93F4889EB47C33A]:0)
   [junit4]    > at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
   [junit4]    > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_816B775345BBAEC2-001
   [junit4]   2> Jul 05, 2017 11:22:42 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=FST50, a_t=BlockTreeOrds(blocksize=128), a_i=FST50, id=FST50}, docValues:{_version_=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1128, maxMBSortInHeap=5.897491505619646, sim=RandomSimilarity(queryNorm=false): {}, locale=en, timezone=Africa/Ceuta
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=138995144,total=526909440
   [junit4]   2> NOTE: All tests run in this JVM: [DateFieldTest, TestDownShardTolerantSearch, MinimalSchemaTest, TimeZoneUtilsTest, AutoCommitTest, TestTolerantSearch, TestStressReorder, DirectUpdateHandlerTest, TestBlobHandler, BasicAuthIntegrationTest, DistributedQueueTest, TestSolrCloudWithSecureImpersonation, BasicDistributedZk2Test, CloudMLTQParserTest, ConfigureRecoveryStrategyTest, NumericFieldsTest, HdfsSyncSliceTest, TestExpandComponent, CleanupOldIndexTest, TestSolrCloudWithHadoopAuthPlugin, TestQuerySenderListener, SolrTestCaseJ4Test, SuggesterFSTTest, NodeMutatorTest, SortSpecParsingTest, TestTolerantUpdateProcessorRandomCloud, TestHashQParserPlugin, SimplePostToolTest, BaseCdcrDistributedZkTest, MoveReplicaHDFSTest, RequiredFieldsTest, LeaderElectionTest, TestManagedSchemaThreadSafety, TestPseudoReturnFields, TestRecovery, TestAuthorizationFramework, TestBackupRepositoryFactory, CopyFieldTest, WrapperMergePolicyFactoryTest, HdfsBasicDistributedZk2Test, TestNumericTokenStream, TestReloadDeadlock, ConnectionManagerTest, TestTestInjection, SolrCLIZkUtilsTest, TestSolrConfigHandler, ReplicationFactorTest, TestSSLRandomization, TestMaxScoreQueryParser, TestInfoStreamLogging, TestSubQueryTransformerCrossCore, SolrInfoBeanTest, TemplateUpdateProcessorTest, HdfsChaosMonkeySafeLeaderTest, DateMathParserTest, TestCloudJSONFacetJoinDomain, TestJmxIntegration, TestCustomStream, TestSuggestSpellingConverter, TestSizeLimitedDistributedMap, TestConfigsApi, TestComplexPhraseLeadingWildcard, TestUtils, V2StandaloneTest, TestSchemalessBufferedUpdates, DistributedFacetPivotLongTailTest, TestCryptoKeys, AliasIntegrationTest, TestZkChroot, OverseerTest, TestLocalFSCloudBackupRestore, SpellPossibilityIteratorTest, FullSolrCloudDistribCmdsTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ShowFileRequestHandlerTest, TestAnalyzedSuggestions, TestMultiValuedNumericRangeQuery, UniqFieldsUpdateProcessorFactoryTest, StatsComponentTest, BlockJoinFacetDistribTest, TestSolrCoreProperties, SolrMetricReporterTest, TestExceedMaxTermLength, PropertiesRequestHandlerTest, SuggestComponentContextFilterQueryTest, TestInPlaceUpdatesStandalone, RecoveryAfterSoftCommitTest, PolyFieldTest, TestConfigSetProperties, SimpleFacetsTest, ForceLeaderTest, TestPushWriter, TestCodecSupport, TestRandomFlRTGCloud, LargeFieldTest, ExitableDirectoryReaderTest, TestReversedWildcardFilterFactory, ClassificationUpdateProcessorFactoryTest, TestSurroundQueryParser, TestSha256AuthenticationProvider, AnalysisErrorHandlingTest, ParsingFieldUpdateProcessorsTest, PingRequestHandlerTest, TestReplicaProperties, MultiTermTest, FastVectorHighlighterTest, TestManagedSynonymFilterFactory, DistribCursorPagingTest, TestFastWriter, PrimitiveFieldTypeTest, TestHighFrequencyDictionaryFactory, SearchHandlerTest, TestSolrConfigHandlerConcurrent, DirectUpdateHandlerOptimizeTest, TestClusterStateMutator, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed [330/726 (2!)] on J2 in 83.70s, 1 test, 1 failure <<< FAILURES!

[...truncated 44523 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1347 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1347/

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.update.TestHdfsUpdateLog

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([71DD60B0BD0D738A]:0)




Build Log:
[...truncated 12796 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20170706_124322_0854238295808738570171.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid6160.hprof ...
   [junit4] Heap dump file created [660046704 bytes in 14.766 secs]
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20170706_124322_085772138592570646870.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] WARN: Unhandled exception in event serialization. -> java.lang.OutOfMemoryError: Java heap space
   [junit4] <<< JVM J1: EOF ----

[...truncated 835 lines...]
   [junit4] Suite: org.apache.solr.update.TestHdfsUpdateLog
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestHdfsUpdateLog_71DD60B0BD0D738A-001/init-core-data-001
   [junit4]   2> 956167 WARN  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 956167 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 956169 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 956562 WARN  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 957225 WARN  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 957412 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 957430 WARN  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 957500 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 957546 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_59506_hdfs____f3fkw1/webapp
   [junit4]   2> 958299 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59506
   [junit4]   2> 959098 WARN  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 959103 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 959118 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51001_datanode____m83wxo/webapp
   [junit4]   2> 959498 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51001
   [junit4]   2> 960284 WARN  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 960286 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 960306 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51152_datanode____hzyhl/webapp
   [junit4]   2> 960862 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51152
   [junit4]   2> 961384 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* processReport: from storage DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46 node DatanodeRegistration(127.0.0.1:49950, datanodeUuid=7c56d65d-5c5e-4c4e-b6ef-1172504a9545, infoPort=53005, infoSecurePort=0, ipcPort=47687, storageInfo=lv=-56;cid=testClusterID;nsid=1942765487;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs
   [junit4]   2> 961384 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* processReport: from storage DS-808389ac-305f-4349-bd5b-26eb5df476d5 node DatanodeRegistration(127.0.0.1:49950, datanodeUuid=7c56d65d-5c5e-4c4e-b6ef-1172504a9545, infoPort=53005, infoSecurePort=0, ipcPort=47687, storageInfo=lv=-56;cid=testClusterID;nsid=1942765487;c=0), blocks: 0, hasStaleStorage: false, processing time: 1 msecs
   [junit4]   2> 961386 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* processReport: from storage DS-d1c676bf-e31b-482f-bf21-cabf5e38a453 node DatanodeRegistration(127.0.0.1:37760, datanodeUuid=d0aa286b-d7cb-4288-a8a2-c667f5e9c420, infoPort=57053, infoSecurePort=0, ipcPort=42785, storageInfo=lv=-56;cid=testClusterID;nsid=1942765487;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 961386 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* processReport: from storage DS-b6f9d0de-9965-431f-806e-b36d32210db3 node DatanodeRegistration(127.0.0.1:37760, datanodeUuid=d0aa286b-d7cb-4288-a8a2-c667f5e9c420, infoPort=57053, infoSecurePort=0, ipcPort=42785, storageInfo=lv=-56;cid=testClusterID;nsid=1942765487;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 962371 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 962372 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 962408 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 962427 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 962514 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 962560 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42fb1a67
   [junit4]   2> 962566 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42fb1a67
   [junit4]   2> 962566 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42fb1a67
   [junit4]   2> 962568 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 962598 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 962623 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 962718 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 962729 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 962730 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42fb1a67
   [junit4]   2> 962752 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:48967/solr_hdfs_home
   [junit4]   2> 962752 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 962752 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 962752 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[hdfs://localhost:48967/solr_hdfs_home/collection1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestHdfsUpdateLog_71DD60B0BD0D738A-001/init-core-data-001/]
   [junit4]   2> 962754 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:48967/solr_hdfs_home/collection1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestHdfsUpdateLog_71DD60B0BD0D738A-001/init-core-data-001/snapshot_metadata
   [junit4]   2> 962779 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 962779 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 962779 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 963394 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 963402 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:48967/solr_hdfs_home/collection1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestHdfsUpdateLog_71DD60B0BD0D738A-001/init-core-data-001
   [junit4]   2> 963435 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:48967/solr_hdfs_home/collection1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.update.TestHdfsUpdateLog_71DD60B0BD0D738A-001/init-core-data-001/index
   [junit4]   2> 963444 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 963444 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 963452 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 963453 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=87.62109375, floorSegmentMB=1.1240234375, forceMergeDeletesPctAllowed=28.427669690135527, segmentsPerTier=3.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 963697 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW]]} size 72
   [junit4]   2> 963697 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741825_1001 size 72
   [junit4]   2> 964122 WARN  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 964182 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 964182 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=hdfs://localhost:48967/solr/shard1 defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 964182 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 964201 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 964201 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 964204 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4f6644cf
   [junit4]   2> 964295 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6ec3e484[collection1] main]
   [junit4]   2> 964295 WARN  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 964299 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 964301 INFO  (searcherExecutor-2770-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6ec3e484[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 964302 INFO  (coreLoadExecutor-2769-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572178200288559104
   [junit4]   2> 964305 INFO  (SUITE-TestHdfsUpdateLog-seed#[71DD60B0BD0D738A]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 964342 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFSThreadSafety
   [junit4]   2> 964343 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 964346 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@63c10078[collection1] realtime]
   [junit4]   2> 964346 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 3
   [junit4]   2> 964347 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1572178200335745024,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 964347 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:1572178200335745024
   [junit4]   2> 964385 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 964385 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 964396 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@3213eeb0[collection1] main]
   [junit4]   2> 964396 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 964399 INFO  (searcherExecutor-2770-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3213eeb0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 964399 INFO  (TEST-TestHdfsUpdateLog.testFSThreadSafety-seed#[71DD60B0BD0D738A]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 51
   [junit4]   2> 964457 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[0 (1572178200398659584)]} 0 50
   [junit4]   2> 964472 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[1 (1572178200463671296)]} 0 3
   [junit4]   2> 964485 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[2 (1572178200478351360)]} 0 2
   [junit4]   2> 964497 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[3 (1572178200491982848)]} 0 1
   [junit4]   2> 964511 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[4 (1572178200506662912)]} 0 1
   [junit4]   2> 964523 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5 (1572178200519245824)]} 0 1
   [junit4]   2> 964535 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 964535 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:0
   [junit4]   2> 964535 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[6 (1572178200531828736)]} 0 1
   [junit4]   2> 964607 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 964608 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 964655 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[7 (1572178200544411648)]} 0 108
   [junit4]   2> 964675 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[8 (1572178200675483648)]} 0 5
   [junit4]   2> 964685 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 964686 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 964693 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[9 (1572178200691212288)]} 0 6
   [junit4]   2> 964707 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[10 (1572178200709038080)]} 0 3
   [junit4]   2> 964720 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[11 (1572178200724766720)]} 0 2
   [junit4]   2> 964728 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 964730 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 964733 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[12 (1572178200738398208)]} 0 1
   [junit4]   2> 964748 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[13 (1572178200750981120)]} 0 4
   [junit4]   2> 964758 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 964760 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 964762 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[14 (1572178200767758336)]} 0 3
   [junit4]   2> 964779 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[15 (1572178200782438400)]} 0 6
   [junit4]   2> 964785 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741825_1001 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 964794 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[16 (1572178200800264192)]} 0 4
   [junit4]   2> 964810 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[17 (1572178200815992832)]} 0 4
   [junit4]   2> 964822 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[18 (1572178200831721472)]} 0 2
   [junit4]   2> 964830 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@386d5a6c[collection1] realtime]
   [junit4]   2> 964838 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW]]} size 75
   [junit4]   2> 964839 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW]]} size 75
   [junit4]   2> 964844 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 964845 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 964845 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:0
   [junit4]   2> 964846 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[19 (1572178200845352960)]} 0 12
   [junit4]   2> 964906 INFO  (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@7dfc4197) [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:37760 to delete [blk_1073741825_1001]
   [junit4]   2> 964916 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 964916 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[20 (1572178200869470208)]} 0 60
   [junit4]   2> 964917 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 964929 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[21 (1572178200943919104)]} 0 2
   [junit4]   2> 964941 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[22 (1572178200957550592)]} 0 1
   [junit4]   2> 964954 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[23 (1572178200970133504)]} 0 2
   [junit4]   2> 964965 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 964966 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[24 (1572178200983764992)]} 0 1
   [junit4]   2> 964968 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741835_1011 size 882
   [junit4]   2> 964979 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[25 (1572178200996347904)]} 0 1
   [junit4]   2> 964992 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[26 (1572178201009979392)]} 0 1
   [junit4]   2> 965015 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[27 (1572178201022562304)]} 0 12
   [junit4]   2> 965022 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965023 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965029 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[28 (1572178201046679552)]} 0 3
   [junit4]   2> 965041 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[29 (1572178201062408192)]} 0 1
   [junit4]   2> 965053 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[30 (1572178201074991104)]} 0 1
   [junit4]   2> 965067 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965068 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965069 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[31 (1572178201087574016)]} 0 4
   [junit4]   2> 965080 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[32 (1572178201103302656)]} 0 1
   [junit4]   2> 965092 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[33 (1572178201115885568)]} 0 1
   [junit4]   2> 965104 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[34 (1572178201128468480)]} 0 1
   [junit4]   2> 965116 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[35 (1572178201141051392)]} 0 1
   [junit4]   2> 965130 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[36 (1572178201155731456)]} 0 1
   [junit4]   2> 965142 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965144 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[37 (1572178201168314368)]} 0 2
   [junit4]   2> 965144 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741838_1014 size 384
   [junit4]   2> 965156 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[38 (1572178201181945856)]} 0 1
   [junit4]   2> 965168 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[39 (1572178201194528768)]} 0 1
   [junit4]   2> 965180 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[40 (1572178201207111680)]} 0 1
   [junit4]   2> 965191 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[41 (1572178201219694592)]} 0 1
   [junit4]   2> 965203 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[42 (1572178201232277504)]} 0 1
   [junit4]   2> 965215 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[43 (1572178201244860416)]} 0 1
   [junit4]   2> 965227 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[44 (1572178201257443328)]} 0 1
   [junit4]   2> 965229 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965230 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965239 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[45 (1572178201270026240)]} 0 1
   [junit4]   2> 965252 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[46 (1572178201281560576)]} 0 2
   [junit4]   2> 965264 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[47 (1572178201296240640)]} 0 1
   [junit4]   2> 965275 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[48 (1572178201307774976)]} 0 1
   [junit4]   2> 965299 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[49 (1572178201320357888)]} 0 13
   [junit4]   2> 965310 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 0
   [junit4]   2> 965312 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741840_1016 size 795
   [junit4]   2> 965321 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[50 (1572178201352863744)]} 0 3
   [junit4]   2> 965346 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[51 (1572178201368592384)]} 0 14
   [junit4]   2> 965359 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[52 (1572178201395855360)]} 0 1
   [junit4]   2> 965371 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[53 (1572178201408438272)]} 0 1
   [junit4]   2> 965372 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@21ebc70c[collection1] realtime]
   [junit4]   2> 965383 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW]]} size 75
   [junit4]   2> 965384 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW]]} size 75
   [junit4]   2> 965385 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 965385 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 965385 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:0
   [junit4]   2> 965386 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[54 (1572178201421021184)]} 0 4
   [junit4]   2> 965456 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965457 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965485 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[55 (1572178201436749824)]} 0 88
   [junit4]   2> 965497 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[56 (1572178201540558848)]} 0 0
   [junit4]   2> 965499 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965501 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965509 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[57 (1572178201553141760)]} 0 1
   [junit4]   2> 965523 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[58 (1572178201567821824)]} 0 1
   [junit4]   2> 965524 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965530 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965536 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[59 (1572178201580404736)]} 0 1
   [junit4]   2> 965547 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[60 (1572178201592987648)]} 0 1
   [junit4]   2> 965560 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[61 (1572178201605570560)]} 0 2
   [junit4]   2> 965566 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965567 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741845_1021 size 221
   [junit4]   2> 965574 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[62 (1572178201618153472)]} 0 3
   [junit4]   2> 965586 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[63 (1572178201633882112)]} 0 1
   [junit4]   2> 965601 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[64 (1572178201649610752)]} 0 1
   [junit4]   2> 965604 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965605 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965613 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[65 (1572178201662193664)]} 0 1
   [junit4]   2> 965630 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965630 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[66 (1572178201677922304)]} 0 3
   [junit4]   2> 965631 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965643 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[67 (1572178201692602368)]} 0 2
   [junit4]   2> 965656 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[68 (1572178201707282432)]} 0 1
   [junit4]   2> 965661 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965661 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965668 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[69 (1572178201719865344)]} 0 1
   [junit4]   2> 965669 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741828_1004 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 965684 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[70 (1572178201731399680)]} 0 6
   [junit4]   2> 965696 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[71 (1572178201749225472)]} 0 1
   [junit4]   2> 965698 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2fdbc252[collection1] realtime]
   [junit4]   2> 965710 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 76
   [junit4]   2> 965710 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 76
   [junit4]   2> 965711 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 965711 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 965711 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:0
   [junit4]   2> 965712 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[72 (1572178201761808384)]} 0 5
   [junit4]   2> 965747 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965747 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965774 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[73 (1572178201778585600)]} 0 51
   [junit4]   2> 965778 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741851_1027{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965783 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741851_1027 size 221
   [junit4]   2> 965786 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[74 (1572178201843597312)]} 0 1
   [junit4]   2> 965802 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[75 (1572178201856180224)]} 0 5
   [junit4]   2> 965814 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[76 (1572178201871908864)]} 0 1
   [junit4]   2> 965825 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[77 (1572178201884491776)]} 0 1
   [junit4]   2> 965830 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 0
   [junit4]   2> 965830 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965837 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[78 (1572178201897074688)]} 0 1
   [junit4]   2> 965849 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[79 (1572178201909657600)]} 0 1
   [junit4]   2> 965861 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965862 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[80 (1572178201922240512)]} 0 1
   [junit4]   2> 965862 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965875 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965875 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[81 (1572178201936920576)]} 0 1
   [junit4]   2> 965876 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 965887 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[82 (1572178201949503488)]} 0 1
   [junit4]   2> 965903 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[83 (1572178201962086400)]} 0 4
   [junit4]   2> 965915 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[84 (1572178201977815040)]} 0 1
   [junit4]   2> 965918 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741855_1031{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965919 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741855_1031{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 965934 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[85 (1572178201991446528)]} 0 7
   [junit4]   2> 965945 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[86 (1572178202010320896)]} 0 1
   [junit4]   2> 965957 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[87 (1572178202022903808)]} 0 1
   [junit4]   2> 965971 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[88 (1572178202035486720)]} 0 3
   [junit4]   2> 965974 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965975 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 965986 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[89 (1572178202051215360)]} 0 3
   [junit4]   2> 966000 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[90 (1572178202066944000)]} 0 2
   [junit4]   2> 966001 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966006 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741857_1033 size 384
   [junit4]   2> 966012 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[91 (1572178202080575488)]} 0 1
   [junit4]   2> 966028 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[92 (1572178202096304128)]} 0 1
   [junit4]   2> 966033 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966034 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741858_1034 size 421
   [junit4]   2> 966039 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[93 (1572178202108887040)]} 0 0
   [junit4]   2> 966051 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[94 (1572178202121469952)]} 0 1
   [junit4]   2> 966062 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[95 (1572178202133004288)]} 0 1
   [junit4]   2> 966074 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[96 (1572178202145587200)]} 0 0
   [junit4]   2> 966079 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966081 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966086 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[97 (1572178202158170112)]} 0 1
   [junit4]   2> 966086 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741833_1009 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966101 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[98 (1572178202169704448)]} 0 4
   [junit4]   2> 966110 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f24f5ef[collection1] realtime]
   [junit4]   2> 966131 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 76
   [junit4]   2> 966131 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 76
   [junit4]   2> 966132 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 966132 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 966132 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:0
   [junit4]   2> 966178 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966180 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966180 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[99 (1572178202192773120)]} 0 62
   [junit4]   2> 966192 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[100 (1572178202269319168)]} 0 1
   [junit4]   2> 966204 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[101 (1572178202281902080)]} 0 0
   [junit4]   2> 966227 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[102 (1572178202306019328)]} 0 1
   [junit4]   2> 966231 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966235 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966241 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[103 (1572178202318602240)]} 0 3
   [junit4]   2> 966255 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[104 (1572178202335379456)]} 0 1
   [junit4]   2> 966262 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966264 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966266 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[105 (1572178202346913792)]} 0 1
   [junit4]   2> 966278 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[106 (1572178202359496704)]} 0 1
   [junit4]   2> 966290 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[107 (1572178202372079616)]} 0 1
   [junit4]   2> 966293 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966293 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966302 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[108 (1572178202384662528)]} 0 1
   [junit4]   2> 966313 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[109 (1572178202396196864)]} 0 1
   [junit4]   2> 966314 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW]]} size 0
   [junit4]   2> 966314 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966325 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[110 (1572178202408779776)]} 0 1
   [junit4]   2> 966337 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741866_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966337 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[111 (1572178202421362688)]} 0 1
   [junit4]   2> 966339 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741866_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966349 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[112 (1572178202433945600)]} 0 1
   [junit4]   2> 966356 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741867_1043{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966356 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741867_1043{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966361 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[113 (1572178202446528512)]} 0 1
   [junit4]   2> 966362 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741841_1017 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966364 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741831_1007 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966366 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741843_1019 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966368 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966372 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[114 (1572178202458062848)]} 0 1
   [junit4]   2> 966379 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741837_1013 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966381 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741838_1014 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966385 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[115 (1572178202470645760)]} 0 2
   [junit4]   2> 966385 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741832_1008 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966389 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741840_1016 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966391 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741845_1021 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966393 INFO  (IPC Server handler 7 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741835_1011 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966395 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741830_1006 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966396 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[116 (1572178202483228672)]} 0 0
   [junit4]   2> 966397 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741839_1015 127.0.0.1:37760 127.0.0.1:49950
   [junit4]   2> 966405 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741846_1022 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966408 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[117 (1572178202495811584)]} 0 1
   [junit4]   2> 966442 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[118 (1572178202507345920)]} 0 23
   [junit4]   2> 966443 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f05239a[collection1] realtime]
   [junit4]   2> 966448 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 76
   [junit4]   2> 966449 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 76
   [junit4]   2> 966452 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741827_1003 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966453 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741829_1005 127.0.0.1:49950 127.0.0.1:37760
   [junit4]   2> 966453 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 966453 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 966453 INFO  (commitScheduler-2773-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@31b36a01 commitCommandVersion:0
   [junit4]   2> 966471 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[119 (1572178202542997504)]} 0 18
   [junit4]   2> 966474 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966474 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966482 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[120 (1572178202573406208)]} 0 0
   [junit4]   2> 966485 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741870_1046{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966486 INFO  (IPC Server handler 1 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741870_1046{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966494 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[121 (1572178202585989120)]} 0 1
   [junit4]   2> 966505 INFO  (IPC Server handler 3 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741871_1047{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966506 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[122 (1572178202598572032)]} 0 1
   [junit4]   2> 966511 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741871_1047 size 1008
   [junit4]   2> 966517 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[123 (1572178202610106368)]} 0 1
   [junit4]   2> 966525 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741872_1048{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED]]} size 0
   [junit4]   2> 966525 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741872_1048{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|FINALIZED], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966534 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[124 (1572178202623737856)]} 0 4
   [junit4]   2> 966545 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[125 (1572178202639466496)]} 0 1
   [junit4]   2> 966548 INFO  (IPC Server handler 6 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741873_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966554 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741873_1049 size 421
   [junit4]   2> 966557 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[126 (1572178202652049408)]} 0 0
   [junit4]   2> 966567 INFO  (IPC Server handler 8 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741874_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966569 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[127 (1572178202663583744)]} 0 1
   [junit4]   2> 966569 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741874_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966580 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[128 (1572178202676166656)]} 0 1
   [junit4]   2> 966586 INFO  (IPC Server handler 0 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741875_1051{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4]   2> 966587 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741875_1051 size 221
   [junit4]   2> 966591 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[129 (1572178202688749568)]} 0 0
   [junit4]   2> 966603 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[130 (1572178202700283904)]} 0 1
   [junit4]   2> 966614 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[131 (1572178202712866816)]} 0 0
   [junit4]   2> 966615 INFO  (IPC Server handler 5 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741876_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 0
   [junit4]   2> 966616 INFO  (IPC Server handler 4 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741876_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW], ReplicaUC[[DISK]DS-d1c676bf-e31b-482f-bf21-cabf5e38a453:NORMAL:127.0.0.1:37760|RBW]]} size 0
   [junit4]   2> 966626 INFO  (Thread-49591) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[132 (1572178202724401152)]} 0 1
   [junit4]   2> 966630 INFO  (IPC Server handler 2 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37760 is added to blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-808389ac-305f-4349-bd5b-26eb5df476d5:NORMAL:127.0.0.1:49950|RBW]]} size 0
   [junit4]   2> 966630 INFO  (IPC Server handler 9 on 48967) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49950 is added to blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b6f9d0de-9965-431f-806e-b36d32210db3:NORMAL:127.0.0.1:37760|RBW], ReplicaUC[[DISK]DS-bb622c97-8f7c-4beb-9ddf-dafbf719dc46:NORMAL:127.0.0.1:49950|FINALIZED]]} size 0
   [junit4

[...truncated too long message...]

-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/byte-buddy-1.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mockito-core-2.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/objenesis-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-8.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/contrib/analysis-extras/lib/icu4j-59.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20170706_124322_0851540403983220184781.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20170706_124322_0854203761868672364522.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:754: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:267: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:549: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1455: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:985: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 327 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No artifacts from Lucene-Solr-NightlyTests-master #1346 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1348 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1348/

6 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([6365789E74A119D0]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexSorting

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([6365789E74A119D0]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([DB34AC9D0CDE7527:53609347A22218DF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:     1) Thread[id=9592, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=9591, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:
   1) Thread[id=9592, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=9591, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
        at __randomizedtesting.SeedInfo.seed([DB34AC9D0CDE7527]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=9592, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    2) Thread[id=9591, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=9592, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   2) Thread[id=9591, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
        at __randomizedtesting.SeedInfo.seed([DB34AC9D0CDE7527]:0)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 8 in https://127.0.0.1:38984/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 8 in https://127.0.0.1:38984/solr
        at __randomizedtesting.SeedInfo.seed([DB34AC9D0CDE7527:1AC4D531218EBF80]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:868)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:589)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1906 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> júl 07, 2017 7:20:28 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexSorting
   [junit4]   2>    1) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    3) Thread[id=14587, name=TEST-TestIndexSorting.testRandom3-seed#[6365789E74A119D0], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at sun.nio.ch.NativeThread.current(Native Method)
   [junit4]   2>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:284)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:50)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterSeekableByteChannel.position(FilterSeekableByteChannel.java:74)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterSeekableByteChannel.position(FilterSeekableByteChannel.java:74)
   [junit4]   2>         at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(SimpleFSDirectory.java:171)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
   [junit4]   2>         at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
   [junit4]   2>         at org.apache.lucene.store.DataInput.readLong(DataInput.java:157)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.readLong(BufferedIndexInput.java:196)
   [junit4]   2>         at org.apache.lucene.store.MockIndexInputWrapper.readLong(MockIndexInputWrapper.java:171)
   [junit4]   2>         at org.apache.lucene.store.IndexInput$1.readLong(IndexInput.java:150)
   [junit4]   2>         at org.apache.lucene.util.packed.DirectReader$DirectPackedReader64.get(DirectReader.java:343)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70DocValuesProducer$22.nextValue(Lucene70DocValuesProducer.java:1271)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer$3$1.nextValue(DocValuesConsumer.java:446)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValues(Lucene70DocValuesConsumer.java:162)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedNumericField(Lucene70DocValuesConsumer.java:530)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:371)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:143)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4390)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4030)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2233)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5048)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1718)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1460)
   [junit4]   2>         at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2301)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    4) Thread[id=14586, name=SUITE-TestIndexSorting-seed#[6365789E74A119D0], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.method=testRandom3 -Dtests.seed=6365789E74A119D0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=sk -Dtests.timezone=America/Guayaquil -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7199s J2 | TestIndexSorting.testRandom3 <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([6365789E74A119D0]:0)
   [junit4]   2> júl 07, 2017 7:20:28 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> júl 07, 2017 7:20:48 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexSorting:
   [junit4]   2>    1) Thread[id=14587, name=TEST-TestIndexSorting.testRandom3-seed#[6365789E74A119D0], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesProducer$7.setDocument(DirectDocValuesProducer.java:489)
   [junit4]   2>         at org.apache.lucene.index.LegacySortedNumericDocValuesWrapper.nextDoc(LegacySortedNumericDocValuesWrapper.java:53)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer$SortedNumericDocValuesSub.nextDoc(DocValuesConsumer.java:359)
   [junit4]   2>         at org.apache.lucene.index.DocIDMerger$SortedDocIDMerger.next(DocIDMerger.java:176)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer$3$1.nextDoc(DocValuesConsumer.java:414)
   [junit4]   2>         at org.apache.lucene.codecs.LegacyDocValuesIterables$6$1.next(LegacyDocValuesIterables.java:295)
   [junit4]   2>         at org.apache.lucene.codecs.LegacyDocValuesIterables$6$1.next(LegacyDocValuesIterables.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer$1$1.next(DirectDocValuesConsumer.java:330)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer$1$1.next(DirectDocValuesConsumer.java:315)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericFieldValues(DirectDocValuesConsumer.java:119)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addSortedNumericField(DirectDocValuesConsumer.java:262)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:371)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:143)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4390)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4030)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2233)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5048)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1718)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1460)
   [junit4]   2>         at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2301)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> júl 07, 2017 7:20:48 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=14587, name=TEST-TestIndexSorting.testRandom3-seed#[6365789E74A119D0], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexSorting_6365789E74A119D0-001
   [junit4]   2> Jul 07, 2017 12:20:48 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=Lucene50(blocksize=128)}, docValues:{multi_valued_double=DocValuesFormat(name=Lucene70), multi_valued_long=DocValuesFormat(name=Direct), bytes=DocValuesFormat(name=Lucene70), double=DocValuesFormat(name=Lucene70), foo=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Lucene70), multi_valued_int=DocValuesFormat(name=Memory), float=DocValuesFormat(name=Asserting), multi_valued_bytes=DocValuesFormat(name=Lucene70), multi_valued_float=DocValuesFormat(name=Direct), int=DocValuesFormat(name=Lucene70), long=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=266, maxMBSortInHeap=5.045152472310973, sim=RandomSimilarity(queryNorm=false): {id=DFR I(ne)1}, locale=sk, timezone=America/Guayaquil
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=164175384,total=444596224
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterMergePolicy, TestTermScorer, TestIntArrayDocIdSet, Test2BBinaryDocValues, TestDirectoryReader, TestMathUtil, TestMultiTermsEnum, TestStressIndexing2, TestAttributeSource, TestTermVectors, TestSegmentTermDocs, TestFieldReuse, TestDemoParallelLeafReader, TestForUtil, TestRegexpQuery, TestSimpleFSDirectory, TestVersion, TestFieldsReader, TestBoolean2ScorerSupplier, TestRamUsageEstimator, TestSortedSetSortField, TestSpanTermQuery, TestSetOnce, TestSpanNearQuery, TestMixedCodecs, TestFilterDirectory, Test2BPagedBytes, TestPrefixQuery, TestConjunctions, TestMultiFields, TestStressAdvance, TestLucene70NormsFormat, TestMultiset, TestSloppyMath, TestSleepingLockWrapper, TestIndexWriterMerging, TestBufferedIndexInput, TestIndexWriterMaxDocs, TestFieldCacheRewriteMethod, TestRollingBuffer, MultiCollectorTest, TestComplexExplanationsOfNonMatches, TestSearchForDuplicates, TestCharacterUtils, TestTimSorterWorstCase, TestPackedInts, TestIntsRef, TestNIOFSDirectory, TestSpanBoostQuery, TestIndexCommit, TestField, TestFastCompressionMode, TestShardSearching, TestBlockPostingsFormat, TestTopDocsCollector, TestNRTReaderWithThreads, TestMutablePointsReaderUtils, Test2BSortedDocValuesOrds, TestPolygon2D, TestIndexManyDocuments, TestSpans, TestIOUtils, TestEarlyTerminatingSortingCollector, TestConjunctionDISI, TestDisjunctionMaxQuery, TestLongPostings, TestPayloads, TestByteBlockPool, TestLucene50StoredFieldsFormatHighCompression, TestSortRescorer, TestSpanSearchEquivalence, TestMultiLevelSkipList, TestCachingTokenFilter, TestIndexSearcher, TestMinShouldMatch2, TestAutomaton, TestPerFieldDocValuesFormat, TestIndexWriterReader, TestFSTs, TestMultiMMap, TestBooleanOr, TestBasics, TestNRTThreads, TestDuelingCodecs, TestAtomicUpdate, TestStandardAnalyzer, TestIndexWriterCommit, TestTimeLimitingCollector, TestSearchWithThreads, TestSearchAfter, TestConcurrentMergeScheduler, TestHighCompressionMode, TestTopDocsMerge, TestStressIndexing, TestRollingUpdates, TestSnapshotDeletionPolicy, TestTieredMergePolicy, TestIndexWriterForceMerge, TestPerFieldPostingsFormat, TestArrayUtil, TestLevenshteinAutomata, TestByteSlices, TestConsistentFieldNumbers, TestSimpleExplanations, TestLockFactory, TestSegmentMerger, TestPhraseQuery, TestCollectionUtil, TestBagOfPositions, TestFixedBitSet, TestComplexExplanations, TestRegexpRandom2, TestSegmentReader, TestParallelCompositeReader, TestPagedBytes, TestTransactions, TestCrash, TestPerFieldPostingsFormat2, TestSort, TestToken, TestSimpleSearchEquivalence, TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, TestBooleanMinShouldMatch, TestTermsEnum2, TestDeterminizeLexicon, TestIndexWriterOnDiskFull, TestCustomNorms, TestUTF32ToUTF8, TestSpanExplanations, TestLazyProxSkipping, TestDeterminism, TestMergeSchedulerExternal, TestBagOfPostings, TestTermVectorsWriter, TestCodecs, TestCharTermAttributeImpl, TestBytesRefHash, TestNearSpansOrdered, TestSameScoresWithThreads, TestTermRangeQuery, TestCrashCausesCorruptIndex, TestScorerPerf, TestLiveFieldValues, TestForTooMuchCloning, TestFuzzyQuery, TestDoc, TestWildcard, TestSimilarity2, TestExternalCodecs, TestNoDeletionPolicy, TestSimpleExplanationsOfNonMatches, TestTermVectorsReader, TestPerSegmentDeletes, TestPrefixInBooleanQuery, TestSegmentTermEnum, TestUniqueTermCount, TestUnicodeUtil, TestSortRandom, TestPrefixCodedTerms, TestAllFilesHaveCodecHeader, TestMaxTermFrequency, TestIndexableField, TestSentinelIntSet, TestCheckIndex, TestPayloadsOnVectors, TestCompiledAutomaton, TestSubScorerFreqs, TestMultiTermQueryRewrites, TestDateSort, TestSpanExplanationsOfNonMatches, TestSmallFloat, TestDateTools, TestSameTokenSamePosition, TestCachingCollector, TestTotalHitCountCollector, TestNoMergePolicy, TestIndexWriterLockRelease, TestTwoPhaseCommitTool, TestCodecHoldsOpenFiles, TestSimpleAttributeImpl, Test2BPositions, Test2BPostingsBytes, TestByteArrayDataInput, TestCharArrayMap, TestDelegatingAnalyzerWrapper, TestPackedTokenAttributeImpl, TestLucene50TermVectorsFormat, TestExceedMaxTermLength, TestFieldInvertState, TestFilterCodecReader, TestFilterDirectoryReader, TestIndexSorting]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.seed=6365789E74A119D0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=sk -Dtests.timezone=America/Guayaquil -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestIndexSorting (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([6365789E74A119D0]:0)
   [junit4] Completed [454/454 (1!)] on J2 in 7220.16s, 15 tests, 2 errors <<< FAILURES!

[...truncated 9946 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/init-core-data-001
   [junit4]   2> 630700 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 630701 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 630703 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001
   [junit4]   2> 630703 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 630703 INFO  (Thread-852) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 630703 INFO  (Thread-852) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 630711 ERROR (Thread-852) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 630803 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:59546
   [junit4]   2> 630811 INFO  (jetty-launcher-494-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 630811 INFO  (jetty-launcher-494-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 630814 INFO  (jetty-launcher-494-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36b787f0{/solr,null,AVAILABLE}
   [junit4]   2> 630814 INFO  (jetty-launcher-494-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d5e3118{/solr,null,AVAILABLE}
   [junit4]   2> 630816 INFO  (jetty-launcher-494-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@171ea772{SSL,[ssl, http/1.1]}{127.0.0.1:50242}
   [junit4]   2> 630816 INFO  (jetty-launcher-494-thread-2) [    ] o.e.j.s.Server Started @666461ms
   [junit4]   2> 630816 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50242}
   [junit4]   2> 630816 ERROR (jetty-launcher-494-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 630816 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 630816 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 630816 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 630817 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-07T14:28:59.325Z
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@57fd4143{SSL,[ssl, http/1.1]}{127.0.0.1:38984}
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.e.j.s.Server Started @666463ms
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38984}
   [junit4]   2> 630818 ERROR (jetty-launcher-494-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 630818 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-07T14:28:59.326Z
   [junit4]   2> 630819 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 630827 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 630830 INFO  (jetty-launcher-494-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59546/solr
   [junit4]   2> 630835 INFO  (jetty-launcher-494-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59546/solr
   [junit4]   2> 630939 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 630941 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50242_solr
   [junit4]   2> 630942 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.c.Overseer Overseer (id=98267151676997636-127.0.0.1:50242_solr-n_0000000000) starting
   [junit4]   2> 630946 INFO  (jetty-launcher-494-thread-1) [n:127.0.0.1:38984_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 630948 INFO  (jetty-launcher-494-thread-1) [n:127.0.0.1:38984_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38984_solr
   [junit4]   2> 630954 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50242_solr
   [junit4]   2> 630956 INFO  (zkCallback-502-thread-1-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 630956 INFO  (zkCallback-506-thread-1-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 630961 INFO  (zkCallback-506-thread-2-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 630961 INFO  (OverseerStateUpdate-98267151676997636-127.0.0.1:50242_solr-n_0000000000) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 631025 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50242.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 631025 INFO  (jetty-launcher-494-thread-1) [n:127.0.0.1:38984_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38984.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 631030 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50242.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 631031 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50242.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 631031 INFO  (jetty-launcher-494-thread-1) [n:127.0.0.1:38984_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38984.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 631031 INFO  (jetty-launcher-494-thread-1) [n:127.0.0.1:38984_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38984.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 631032 INFO  (jetty-launcher-494-thread-2) [n:127.0.0.1:50242_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001/node2/.
   [junit4]   2> 631032 INFO  (jetty-launcher-494-thread-1) [n:127.0.0.1:38984_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001/node1/.
   [junit4]   2> 631078 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 631080 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59546/solr ready
   [junit4]   2> 631083 INFO  (SUITE-TestTlogReplica-seed#[DB34AC9D0CDE7527]-worker) [    ] o.a.s.c.TestTlogReplica Using legacyCloud?: true
   [junit4]   2> 631111 INFO  (qtp1310427979-3656) [n:127.0.0.1:38984_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=true&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 631112 INFO  (qtp1310427979-3656) [n:127.0.0.1:38984_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=true&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 631131 INFO  (TEST-TestTlogReplica.testRealTimeGet-seed#[DB34AC9D0CDE7527]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRealTimeGet
   [junit4]   2> 631133 INFO  (qtp1310427979-3657) [n:127.0.0.1:38984_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 631135 INFO  (OverseerThreadFactory-1092-thread-1-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_real_time_get
   [junit4]   2> 631246 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 631247 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 631274 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 631275 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 631384 INFO  (zkCallback-506-thread-2-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 631384 INFO  (zkCallback-502-thread-2-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 632277 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 632287 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 632290 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 632290 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_t2' using configuration from collection tlog_replica_test_real_time_get, trusted=true
   [junit4]   2> 632291 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38984.solr.core.tlog_replica_test_real_time_get.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 632291 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 632291 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_t2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001/node1/tlog_replica_test_real_time_get_shard1_replica_t2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001/node1/./tlog_replica_test_real_time_get_shard1_replica_t2/data/]
   [junit4]   2> 632292 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 632300 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 632303 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 632303 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_t1' using configuration from collection tlog_replica_test_real_time_get, trusted=true
   [junit4]   2> 632303 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50242.solr.core.tlog_replica_test_real_time_get.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@c3051e7
   [junit4]   2> 632304 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 632304 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_t1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001/node2/tlog_replica_test_real_time_get_shard1_replica_t1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_DB34AC9D0CDE7527-001/tempDir-001/node2/./tlog_replica_test_real_time_get_shard1_replica_t1/data/]
   [junit4]   2> 632366 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 632366 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 632368 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 632368 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 632371 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 632371 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 632372 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3b3dc234[tlog_replica_test_real_time_get_shard1_replica_t1] main]
   [junit4]   2> 632373 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 632373 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 632373 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 632374 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 632374 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 632375 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@4da0e4ed[tlog_replica_test_real_time_get_shard1_replica_t2] main]
   [junit4]   2> 632376 INFO  (searcherExecutor-1098-thread-1-processing-n:127.0.0.1:50242_solr x:tlog_replica_test_real_time_get_shard1_replica_t1 s:shard1 c:tlog_replica_test_real_time_get) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t1] Registered new searcher Searcher@3b3dc234[tlog_replica_test_real_time_get_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 632376 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572274428585181184
   [junit4]   2> 632378 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 632378 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 632386 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 632387 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 632388 INFO  (searcherExecutor-1097-thread-1-processing-n:127.0.0.1:38984_solr x:tlog_replica_test_real_time_get_shard1_replica_t2 s:shard1 c:tlog_replica_test_real_time_get) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t2] Registered new searcher Searcher@4da0e4ed[tlog_replica_test_real_time_get_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 632388 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1572274428597764096
   [junit4]   2> 632487 INFO  (zkCallback-506-thread-2-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 632487 INFO  (zkCallback-502-thread-2-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 632887 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 632887 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 632887 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 632887 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t1 url=https://127.0.0.1:50242/solr START replicas=[https://127.0.0.1:38984/solr/tlog_replica_test_real_time_get_shard1_replica_t2/] nUpdates=100
   [junit4]   2> 632888 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t1 url=https://127.0.0.1:50242/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 632891 INFO  (qtp1310427979-3650) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 632892 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 632892 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 632892 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 632892 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 632897 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/ shard1
   [junit4]   2> 632999 INFO  (zkCallback-506-thread-2-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 632999 INFO  (zkCallback-502-thread-2-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 633048 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 633051 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1776
   [junit4]   2> 633155 INFO  (zkCallback-502-thread-2-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 633155 INFO  (zkCallback-506-thread-2-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 633396 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t2 starting background replication from leader
   [junit4]   2> 633396 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 633400 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 633400 INFO  (indexFetcher-1108-thread-1) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 633403 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2156
   [junit4]   2> 633405 INFO  (qtp1451090140-3648) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 633405 INFO  (indexFetcher-1108-thread-1) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 633405 INFO  (indexFetcher-1108-thread-1) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 633405 INFO  (indexFetcher-1108-thread-1) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 633405 INFO  (indexFetcher-1108-thread-1) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1  x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 633407 INFO  (qtp1310427979-3657) [n:127.0.0.1:38984_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 633507 INFO  (zkCallback-502-thread-2-processing-n:127.0.0.1:50242_solr) [n:127.0.0.1:50242_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 633507 INFO  (zkCallback-506-thread-2-processing-n:127.0.0.1:38984_solr) [n:127.0.0.1:38984_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 634408 INFO  (qtp1310427979-3657) [n:127.0.0.1:38984_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3274
   [junit4]   2> 634424 INFO  (qtp1310427979-3654) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2}{add=[0 (1572274430728470528)]} 0 0
   [junit4]   2> 634425 INFO  (qtp1451090140-3649) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38984/solr/tlog_replica_test_real_time_get_shard1_replica_t2/&wt=javabin&version=2}{add=[0 (1572274430728470528)]} 0 5
   [junit4]   2> 634425 INFO  (qtp1310427979-3643) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 6
   [junit4]   2> 634426 INFO  (qtp1310427979-3656) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 634430 INFO  (qtp1451090140-3648) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&omitHeader=true&shards.purpose=1&NOW=1499437742934&ids=0&isShard=true&shard.url=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 634431 INFO  (qtp1310427979-3656) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=0&wt=javabin&version=2} status=0 QTime=5
   [junit4]   2> 634432 INFO  (qtp1310427979-3650) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 634432 INFO  (qtp1451090140-3649) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1499437742940&ids=0&isShard=true&shard.url=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 634433 INFO  (qtp1310427979-3650) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 634435 INFO  (qtp1451090140-3644) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 634437 INFO  (qtp1451090140-3655) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1499437742943&ids=0&isShard=true&shard.url=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 634437 INFO  (qtp1451090140-3644) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 634440 INFO  (qtp1310427979-3645) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2}{add=[1 (1572274430747344896)]} 0 0
   [junit4]   2> 634440 INFO  (qtp1451090140-3642) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1572274430747344896)]} 0 1
   [junit4]   2> 634441 INFO  (qtp1451090140-3649) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 634441 INFO  (qtp1451090140-3648) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&omitHeader=true&shards.purpose=1&NOW=1499437742948&ids=1&isShard=true&shard.url=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 634442 INFO  (qtp1451090140-3649) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=1&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 634442 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 634443 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1499437742950&ids=1&isShard=true&shard.url=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 634443 INFO  (qtp1310427979-3652) [n:127.0.0.1:38984_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node1 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 634444 INFO  (qtp1451090140-3653) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:50242/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 634444 INFO  (qtp1451090140-3651) [n:127.0.0.1:50242_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHe

[...truncated too long message...]

       at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), id=Lucene50(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1723, maxMBSortInHeap=6.1244590694894985, sim=RandomSimilarity(queryNorm=true): {}, locale=vi, timezone=America/Nipigon
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=4,free=149312336,total=517472256
   [junit4]   2> NOTE: All tests run in this JVM: [DocumentBuilderTest, TestUtils, SortByFunctionTest, TestManagedResource, DataDrivenBlockJoinTest, TestCloudJSONFacetJoinDomain, TestDownShardTolerantSearch, TestBlobHandler, SpellPossibilityIteratorTest, TestCloudManagedSchema, TestTrie, HardAutoCommitTest, SimpleFacetsTest, ImplicitSnitchTest, WrapperMergePolicyFactoryTest, TestSolrConfigHandlerConcurrent, StatsReloadRaceTest, TestExpandComponent, MetricUtilsTest, LeaderInitiatedRecoveryOnShardRestartTest, PrimitiveFieldTypeTest, BlockDirectoryTest, SignatureUpdateProcessorFactoryTest, CircularListTest, TestDFRSimilarityFactory, BJQParserTest, AtomicUpdatesTest, VMParamsZkACLAndCredentialsProvidersTest, SolrGraphiteReporterTest, UnloadDistributedZkTest, TestCryptoKeys, OverseerTest, TermsComponentTest, TestReplicationHandlerBackup, BasicDistributedZk2Test, TestSubQueryTransformer, TestSuggestSpellingConverter, CopyFieldTest, TestZkChroot, SolrInfoBeanTest, SolrCLIZkUtilsTest, TestFieldCacheReopen, DistributedFacetPivotLongTailTest, RequiredFieldsTest, ExitableDirectoryReaderTest, TestGraphTermsQParserPlugin, TestMinMaxOnMultiValuedField, SuggestComponentContextFilterQueryTest, TestManagedStopFilterFactory, ConfigureRecoveryStrategyTest, HdfsUnloadDistributedZkTest, TestLFUCache, PKIAuthenticationIntegrationTest, EnumFieldTest, TestPseudoReturnFields, DistributedFacetPivotWhiteBoxTest, TestCloudRecovery, TestSolrCLIRunExample, XsltUpdateRequestHandlerTest, CollectionsAPISolrJTest, BadCopyFieldTest, TestJmxIntegration, TestMaxTokenLenTokenizer, TestSolrCloudWithSecureImpersonation, NumericFieldsTest, TestTolerantUpdateProcessorRandomCloud, TestMultiValuedNumericRangeQuery, TestRuleBasedAuthorizationPlugin, PingRequestHandlerTest, V2StandaloneTest, TestStressUserVersions, TestSolrCoreProperties, MoveReplicaHDFSTest, RecoveryZkTest, TestMacroExpander, DateFieldTest, MultiTermTest, HdfsChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsChaosMonkeySafeLeaderTest -Dtests.seed=DB34AC9D0CDE7527 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=vi -Dtests.timezone=America/Nipigon -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | HdfsChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:
   [junit4]    >    1) Thread[id=9592, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    >    2) Thread[id=9591, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DB34AC9D0CDE7527]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=9592, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    2) Thread[id=9591, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DB34AC9D0CDE7527]:0)
   [junit4] Completed [207/727 (2!)] on J2 in 102.52s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 44832 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1349 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1349/

10 tests failed.
FAILED:  org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings

Error Message:
startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=15,endOffset=24,lastStartOffset=18 for field 'dummy'

Stack Trace:
java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=15,endOffset=24,lastStartOffset=18 for field 'dummy'
        at __randomizedtesting.SeedInfo.seed([32EF03E02BE29DAF:58B4BCF172ACBD5C]:0)
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:769)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1460)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
        at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:650)
        at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
        at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:883)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=3539, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=3540, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   1) Thread[id=3539, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=3540, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3539, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    2) Thread[id=3540, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.io.PrintStream.println(PrintStream.java:805)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3539, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   2) Thread[id=3540, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=48245, name=zkCallback-1581-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=48244, name=zkCallback-1581-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=48192, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)    4) Thread[id=48217, name=zkCallback-1581-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=48215, name=StoppableCommitThread-SendThread(127.0.0.1:42281), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    6) Thread[id=48243, name=zkCallback-1581-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=48246, name=zkCallback-1581-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=48216, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=48245, name=zkCallback-1581-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=48244, name=zkCallback-1581-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=48192, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   4) Thread[id=48217, name=zkCallback-1581-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=48215, name=StoppableCommitThread-SendThread(127.0.0.1:42281), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   6) Thread[id=48243, name=zkCallback-1581-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=48246, name=zkCallback-1581-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=48216, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=48245, name=zkCallback-1581-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=48244, name=zkCallback-1581-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=48217, name=zkCallback-1581-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=48215, name=StoppableCommitThread-SendThread(127.0.0.1:42281), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    5) Thread[id=48243, name=zkCallback-1581-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=48246, name=zkCallback-1581-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=48245, name=zkCallback-1581-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=48244, name=zkCallback-1581-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=48217, name=zkCallback-1581-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=48215, name=StoppableCommitThread-SendThread(127.0.0.1:42281), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   5) Thread[id=48243, name=zkCallback-1581-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=48246, name=zkCallback-1581-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03:DCB481A0D3E707FB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03:DCB481A0D3E707FB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03:DCB481A0D3E707FB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:     1) Thread[id=1363, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=1362, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:
   1) Thread[id=1363, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=1362, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1363, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at org.apache.lucene.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:128)         at java.io.PrintStream.write(PrintStream.java:480)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.write(PrintStream.java:527)         at java.io.PrintStream.print(PrintStream.java:669)         at java.io.PrintStream.println(PrintStream.java:823)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)    2) Thread[id=1362, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1363, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at org.apache.lucene.util.TestRuleLimitSysouts$DelegateStream.write(TestRuleLimitSysouts.java:128)
        at java.io.PrintStream.write(PrintStream.java:480)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   2) Thread[id=1362, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:615)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
        at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)




Build Log:
[...truncated 2978 lines...]
   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=false text='\ua8ad\ua89e\ua8ba\ua888\ua8dd v ekpmhl \u30ee\u30b8\u30bf\u30ec\u30d6 \ud928\udf7f\u0a9c qyixp qecVut \ud800\udd7f &m\ud87a\udd04\ue73es'
   [junit4]   2> Exception from random analyzer:
   [junit4]   2> charfilters=
   [junit4]   2>   org.apache.lucene.analysis.charfilter.MappingCharFilter(org.apache.lucene.analysis.charfilter.NormalizeCharMap@5f2b7e9b, java.io.StringReader@94d4039)
   [junit4]   2> tokenizer=
   [junit4]   2>   org.apache.lucene.analysis.standard.StandardTokenizer()
   [junit4]   2> filters=
   [junit4]   2>   org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@3d37a71 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
   [junit4]   2>   org.apache.lucene.analysis.cjk.CJKBigramFilter(ValidatingTokenFilter@64a7d776 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1, 36)
   [junit4]   2>   org.apache.lucene.analysis.en.EnglishPossessiveFilter(ValidatingTokenFilter@1b0bd5b0 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
   [junit4]   2>   org.apache.lucene.analysis.commongrams.CommonGramsQueryFilter(CommonGramsFilter@61b14de term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
   [junit4]   2> offsetsAreCorrect=false
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=32EF03E02BE29DAF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=ar-DZ -Dtests.timezone=PLT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.73s J2 | TestRandomChains.testRandomChainsWithLargeStrings <<<
   [junit4]    > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=15,endOffset=24,lastStartOffset=18 for field 'dummy'
   [junit4]    > at __randomizedtesting.SeedInfo.seed([32EF03E02BE29DAF:58B4BCF172ACBD5C]:0)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:769)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1460)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:650)
   [junit4]    > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
   [junit4]    > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:883)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/common/test/J2/temp/lucene.analysis.core.TestRandomChains_32EF03E02BE29DAF-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {dummy=DFR GB1}, locale=ar-DZ, timezone=PLT
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=212294928,total=287309824
   [junit4]   2> NOTE: All tests run in this JVM: [TestTeeSinkTokenFilter, TestApostropheFilterFactory, TestCJKBigramFilter, TestCustomAnalyzer, TestGermanStemFilter, TestCJKBigramFilterFactory, TestRussianAnalyzer, TestRandomChains]
   [junit4] Completed [77/280 (1!)] on J2 in 92.48s, 2 tests, 1 error <<< FAILURES!

[...truncated 8339 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_54E0BE7A7D1B6A03-001/init-core-data-001
   [junit4]   2> 32647 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=18 numCloses=18
   [junit4]   2> 32648 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 32650 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 32650 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /h/b
   [junit4]   2> 33394 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 34336 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 34641 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 34674 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 34740 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 35031 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_51275_hdfs____na6yvy/webapp
   [junit4]   2> 36095 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51275
   [junit4]   2> 37028 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 37036 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 37064 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53838_datanode____3on5wm/webapp
   [junit4]   2> 37667 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53838
   [junit4]   2> 38433 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 38434 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 38456 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_39281_datanode____yycr1o/webapp
   [junit4]   2> 38842 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[54E0BE7A7D1B6A03]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39281
   [junit4]   2> 39830 INFO  (IPC Server handler 7 on 40718) [    ] BlockStateChange BLOCK* processReport: from storage DS-be2158a4-1552-4baa-a4f7-689270a37b0f node DatanodeRegistration(127.0.0.1:59033, datanodeUuid=8000fa3a-7d29-4be7-bb40-9385da728a2e, infoPort=39484, infoSecurePort=0, ipcPort=33497, storageInfo=lv=-56;cid=testClusterID;nsid=750727030;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 39830 INFO  (IPC Server handler 8 on 40718) [    ] BlockStateChange BLOCK* processReport: from storage DS-90634a69-0aed-4d0d-9f8c-2f61cafbe77d node DatanodeRegistration(127.0.0.1:33800, datanodeUuid=7add9fcd-ddc1-467a-9a93-fd0fda79389f, infoPort=47325, infoSecurePort=0, ipcPort=40303, storageInfo=lv=-56;cid=testClusterID;nsid=750727030;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 39831 INFO  (IPC Server handler 7 on 40718) [    ] BlockStateChange BLOCK* processReport: from storage DS-450342e2-a396-4d20-bf8c-8e376d2db35c node DatanodeRegistration(127.0.0.1:59033, datanodeUuid=8000fa3a-7d29-4be7-bb40-9385da728a2e, infoPort=39484, infoSecurePort=0, ipcPort=33497, storageInfo=lv=-56;cid=testClusterID;nsid=750727030;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 39831 INFO  (IPC Server handler 8 on 40718) [    ] BlockStateChange BLOCK* processReport: from storage DS-ae2ead16-021a-45de-83b5-7c997be4e296 node DatanodeRegistration(127.0.0.1:33800, datanodeUuid=7add9fcd-ddc1-467a-9a93-fd0fda79389f, infoPort=47325, infoSecurePort=0, ipcPort=40303, storageInfo=lv=-56;cid=testClusterID;nsid=750727030;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 39911 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 39912 INFO  (Thread-569) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 39913 INFO  (Thread-569) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 40002 ERROR (Thread-569) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 40012 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.ZkTestServer start zk server on port:36205
   [junit4]   2> 40141 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 40146 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 40149 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 40152 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 40153 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 40155 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 40157 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 40158 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 40161 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 40164 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 40167 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 40186 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 40299 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 40301 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b9ac60d{/h/b,null,AVAILABLE}
   [junit4]   2> 40301 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@390203c2{SSL,[ssl, http/1.1]}{127.0.0.1:53388}
   [junit4]   2> 40302 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.e.j.s.Server Started @42717ms
   [junit4]   2> 40302 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:40718/hdfs__localhost_40718__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_54E0BE7A7D1B6A03-001_tempDir-002_control_data, replicaType=NRT, hostContext=/h/b, hostPort=53388, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_54E0BE7A7D1B6A03-001/control-001/cores}
   [junit4]   2> 40302 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 40302 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 40302 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 40302 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 40302 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-08T10:45:39.390Z
   [junit4]   2> 40314 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 40314 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_54E0BE7A7D1B6A03-001/control-001/solr.xml
   [junit4]   2> 40320 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@41845ee4, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 40333 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36205/solr
   [junit4]   2> 40499 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 40501 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53388_h%2Fb
   [junit4]   2> 40502 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.Overseer Overseer (id=98271935795560453-127.0.0.1:53388_h%2Fb-n_0000000000) starting
   [junit4]   2> 40595 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53388_h%2Fb
   [junit4]   2> 40602 INFO  (OverseerStateUpdate-98271935795560453-127.0.0.1:53388_h%2Fb-n_0000000000) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 40728 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@41845ee4
   [junit4]   2> 40740 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@41845ee4
   [junit4]   2> 40740 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@41845ee4
   [junit4]   2> 40745 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_54E0BE7A7D1B6A03-001/control-001/cores
   [junit4]   2> 41001 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 41004 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[54E0BE7A7D1B6A03]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36205/solr ready
   [junit4]   2> 41074 INFO  (qtp1935431545-797) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:53388_h%252Fb&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 41112 INFO  (OverseerThreadFactory-123-thread-1-processing-n:127.0.0.1:53388_h%2Fb) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 41270 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 41271 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 41388 INFO  (zkCallback-33-thread-1-processing-n:127.0.0.1:53388_h%2Fb) [n:127.0.0.1:53388_h%2Fb    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 42350 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 42375 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 42482 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 42530 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 42530 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@41845ee4
   [junit4]   2> 42537 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40718/solr_hdfs_home
   [junit4]   2> 42537 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 42537 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 42537 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_54E0BE7A7D1B6A03-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:40718/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 42541 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40718/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 42557 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 42557 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 42557 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 43521 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 43528 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40718/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 43568 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40718/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 43575 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 43575 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 43580 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 43582 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=60.53125, floorSegmentMB=1.291015625, forceMergeDeletesPctAllowed=16.225737447333184, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 43851 INFO  (IPC Server handler 3 on 40718) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59033 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ae2ead16-021a-45de-83b5-7c997be4e296:NORMAL:127.0.0.1:33800|RBW], ReplicaUC[[DISK]DS-be2158a4-1552-4baa-a4f7-689270a37b0f:NORMAL:127.0.0.1:59033|RBW]]} size 72
   [junit4]   2> 43852 INFO  (IPC Server handler 2 on 40718) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33800 is added to blk_1073741825_1001 size 72
   [junit4]   2> 44258 WARN  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 44338 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 44338 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 44338 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 44363 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 44363 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 44373 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 44473 INFO  (qtp1935431545-794) [n:127.0.0.1:53388_h%2Fb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7668e304[control_collection_shard1_replica_n1] mai

[...truncated too long message...]

l(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=48215, name=StoppableCommitThread-SendThread(127.0.0.1:42281), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    6) Thread[id=48243, name=zkCallback-1581-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    7) Thread[id=48246, name=zkCallback-1581-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    8) Thread[id=48216, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=48245, name=zkCallback-1581-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    2) Thread[id=48244, name=zkCallback-1581-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    3) Thread[id=48217, name=zkCallback-1581-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=48215, name=StoppableCommitThread-SendThread(127.0.0.1:42281), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    5) Thread[id=48243, name=zkCallback-1581-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=48246, name=zkCallback-1581-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([54E0BE7A7D1B6A03]:0)
   [junit4] Completed [725/727 (3!)] on J0 in 127.51s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 43227 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1350 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1350/

9 tests failed.
FAILED:  org.apache.lucene.codecs.lucene70.TestIndexedDISI.testRandom

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit

Error Message:
this IndexWriter is closed

Stack Trace:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
        at org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:230)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
        at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
        at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
        at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
        at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)


FAILED:  org.apache.lucene.search.TestDoubleRangeFieldQueries.testRandomBig

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  org.apache.lucene.util.TestPagedBytes.testDataInputOutput2

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6:8B5FCA82AA91D006]:0)
        at java.util.Arrays.copyOf(Arrays.java:3181)
        at org.apache.lucene.util.PagedBytes.addBlock(PagedBytes.java:158)
        at org.apache.lucene.util.PagedBytes.access$900(PagedBytes.java:36)
        at org.apache.lucene.util.PagedBytes$PagedBytesDataOutput.writeBytes(PagedBytes.java:390)
        at org.apache.lucene.util.TestPagedBytes.testDataInputOutput2(TestPagedBytes.java:116)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)


FAILED:  org.apache.lucene.util.TestPagedBytes.testDataInputOutput

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6:5ED01300913B26B5]:0)
        at java.util.Arrays.copyOf(Arrays.java:3181)
        at org.apache.lucene.util.PagedBytes.addBlock(PagedBytes.java:158)
        at org.apache.lucene.util.PagedBytes.copy(PagedBytes.java:169)
        at org.apache.lucene.util.TestPagedBytes.testDataInputOutput(TestPagedBytes.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:     1) Thread[id=30820, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    2) Thread[id=30818, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=30897, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    4) Thread[id=30923, name=StoppableCommitThread-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    5) Thread[id=30924, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=30892, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    7) Thread[id=30894, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    8) Thread[id=30893, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    9) Thread[id=30895, name=StoppableSearchThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:65)   10) Thread[id=30819, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
   1) Thread[id=30820, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   2) Thread[id=30818, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=30897, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   4) Thread[id=30923, name=StoppableCommitThread-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   5) Thread[id=30924, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=30892, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   7) Thread[id=30894, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   8) Thread[id=30893, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   9) Thread[id=30895, name=StoppableSearchThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:65)
  10) Thread[id=30819, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([243C1376CCE7A90E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=30923, name=StoppableCommitThread-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    2) Thread[id=30892, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    3) Thread[id=30894, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    4) Thread[id=30893, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.common.util.ByteUtils.UTF16toUTF8(ByteUtils.java:90)         at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)         at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    5) Thread[id=30895, name=StoppableSearchThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)         at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)         at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)         at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)         at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)         at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)         at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)         at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)         at org.apache.log4j.Category.callAppenders(Category.java:206)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:897)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)    6) Thread[id=30819, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=30923, name=StoppableCommitThread-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   2) Thread[id=30892, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   3) Thread[id=30894, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   4) Thread[id=30893, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.common.util.ByteUtils.UTF16toUTF8(ByteUtils.java:90)
        at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)
        at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   5) Thread[id=30895, name=StoppableSearchThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:897)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)
   6) Thread[id=30819, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([243C1376CCE7A90E]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([243C1376CCE7A90E:AC682CAC621BC4F6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:222)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 898 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexingSequenceNumbers
   [junit4]   2> Jul 09, 2017 10:03:03 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1072,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
   [junit4]   2> at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4689)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3279)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3411)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3376)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:21 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1071,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:18 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1078,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:13 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1069,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:08 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1075,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:08 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1073,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:38 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1077,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:38 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1074,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:03:13 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1070,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:14:13 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #1735,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at java.util.Arrays.copyOfRange(Arrays.java:3664)
   [junit4]   2> at java.lang.String.<init>(String.java:207)
   [junit4]   2> at java.lang.StringBuilder.toString(StringBuilder.java:407)
   [junit4]   2> at org.apache.lucene.store.BufferedChecksumIndexInput.<init>(BufferedChecksumIndexInput.java:34)
   [junit4]   2> at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:520)
   [junit4]   2> at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.checkIntegrity(Lucene50PostingsReader.java:1283)
   [junit4]   2> at org.apache.lucene.codecs.memory.FSTOrdTermsReader.checkIntegrity(FSTOrdTermsReader.java:875)
   [junit4]   2> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:348)
   [junit4]   2> at org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:286)
   [junit4]   2> at org.apache.lucene.index.MergeReaderWrapper.checkIntegrity(MergeReaderWrapper.java:178)
   [junit4]   2> at org.apache.lucene.index.FilterLeafReader.checkIntegrity(FilterLeafReader.java:408)
   [junit4]   2> at org.apache.lucene.index.SlowCodecReaderWrapper.wrap(SlowCodecReaderWrapper.java:54)
   [junit4]   2> at org.apache.lucene.index.MockRandomMergePolicy$MockRandomOneMerge.wrapForMerge(MockRandomMergePolicy.java:179)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4375)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4030)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
   [junit4]   2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
   [junit4]   2>
   [junit4]   2> Jul 09, 2017 10:14:13 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1076,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]   2> at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]   2> at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]   2> at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]   2> at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]   2> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]   2> at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)
   [junit4]   2>
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexingSequenceNumbers -Dtests.method=testStressConcurrentCommit -Dtests.seed=59EC404FA001EC6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=de-CH -Dtests.timezone=Etc/GMT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1630s J0 | TestIndexingSequenceNumbers.testStressConcurrentCommit <<<
   [junit4]    > Throwable #1: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:230)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1215, name=Thread-1072, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
   [junit4]    > at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4689)
   [junit4]    > at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3279)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3411)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3376)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1216, name=Thread-1073, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1218, name=Thread-1075, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1212, name=Thread-1069, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1213, name=Thread-1070, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1221, name=Thread-1078, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1220, name=Thread-1077, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1715)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:210)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1214, name=Thread-1071, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:896)
   [junit4]    > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:910)
   [junit4]    > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3375)
   [junit4]    > at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:186)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([59EC404FA001EC6]:0)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:624)
   [junit4]    > at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:788)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.proce

[...truncated too long message...]

a:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    3) Thread[id=30894, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    4) Thread[id=30893, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at org.apache.solr.common.util.ByteUtils.UTF16toUTF8(ByteUtils.java:90)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    5) Thread[id=30895, name=StoppableSearchThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4]    >         at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
   [junit4]    >         at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
   [junit4]    >         at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
   [junit4]    >         at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
   [junit4]    >         at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
   [junit4]    >         at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
   [junit4]    >         at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
   [junit4]    >         at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
   [junit4]    >         at org.apache.log4j.Category.callAppenders(Category.java:206)
   [junit4]    >         at org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4]    >         at org.apache.log4j.Category.log(Category.java:856)
   [junit4]    >         at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:897)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
   [junit4]    >         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)
   [junit4]    >    6) Thread[id=30819, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[243C1376CCE7A90E]-SendThread(127.0.0.1:35987), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([243C1376CCE7A90E]:0)
   [junit4] Completed [502/727 (1!)] on J2 in 122.15s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 8199 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid28178.hprof

Total time: 263 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1351 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1351/

No tests ran.

Build Log:
[...truncated 79 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:422: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:459: Ivy is not available

Total time: 0 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1352 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1352/

1 tests failed.
FAILED:  TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml.[empty]

Error Message:


Stack Trace:
Test report file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml was length 0



Build Log:
[...truncated 262 lines...]
   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163200394456052706494.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
   [junit4] at java.security.AccessController.doPrivileged(Native Method)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
   [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4] at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4] at java.io.PrintStream.write(PrintStream.java:482)
   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
   [junit4] at java.io.Writer.write(Writer.java:157)
   [junit4] at java.util.logging.StreamHandler.publish(StreamHandler.java:224)
   [junit4] at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
   [junit4] at java.util.logging.Logger.log(Logger.java:738)
   [junit4] at java.util.logging.Logger.doLog(Logger.java:765)
   [junit4] at java.util.logging.Logger.log(Logger.java:876)
   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException(RandomizedRunner.java:531)
   [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
   [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
   [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
   [junit4] at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:28)
   [junit4] at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
   [junit4] Caused by: java.io.IOException: No space left on device
   [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
   [junit4] at java.io.FileOut
   [junit4] putStream.write(FileOutputStream.java:326)
   [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
   [junit4] ... 26 more
   [junit4] <<< JVM J0: EOF ----

[...truncated 4 lines...]
   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6343353887351313582581.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
   [junit4] at java.security.AccessController.doPrivileged(Native Method)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testRunStarted(RunListenerEmitter.java:40)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testRunStarted(NoExceptionRunListenerDecorator.java:23)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testRunStarted(BeforeAfterRunListenerDecorator.java:24)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$1.notifyListener(OrderedRunNotifier.java:82)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:49)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestRunStarted(OrderedRunNotifier.java:79)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:249)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4] Caused by: java.io.IOException: No space left on device
   [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
   [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
   [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:121)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
   [junit4] at sun.nio.cs.Strea
   [junit4] mEncoder.write(StreamEncoder.java:135)
   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:24)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:31)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeDescription(AbstractEvent.java:119)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEventWithDescription.serialize(AbstractEventWithDescription.java:29)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.SuiteStartedEvent.serialize(SuiteStartedEvent.java:36)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
   [junit4] ... 13 more
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345429632897914706142.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
   [junit4] at java.security.AccessController.doPrivileged(Native Method)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testFailure(RunListenerEmitter.java:54)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testFailure(NoExceptionRunListenerDecorator.java:55)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testFailure(BeforeAfterRunListenerDecorator.java:60)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$4.notifyListener(OrderedRunNotifier.java:129)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestFailure(OrderedRunNotifier.java:126)
   [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$SubNotifier.fireTestFailure(ThreadLeakControl.java:184)
   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:834)
   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:922)
   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverr
   [junit4] idesOnMethodsRule.java:40)
   [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4] at java.lang.Thread.run(Thread.java:748)
   [junit4] Caused by: java.io.IOException: No space left on device
   [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
   [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
   [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:566)
   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.FailureEvent.serialize(FailureEvent.java:46)
   [junit4] at com.carrotsearch.ant.tasks.junit4
   [junit4] .events.Serializer$2.run(Serializer.java:119)
   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
   [junit4] ... 30 more
   [junit4] <<< JVM J2: EOF ----

[...truncated 2 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163348084582467318555.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6166850120803246205.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] at java.lang.Thread.run(Thread.java:748)
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6341196093693912703591.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6348978994367824159899.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] at java.lang.Thread.run(Thread.java:748)
   [junit4] ERROR: JVM J2 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6343973825201621029068.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345414770058120710840.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:754: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build.xml:50: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1455: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:985: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 2 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No artifacts from Lucene-Solr-NightlyTests-master #1349 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1352 - Still Failing

sarowe
Looks like the new lucene2 node has run out of disk space.  I’ll go ask about it on Infra’s hipchat channel.

--
Steve
www.lucidworks.com

> On Jul 12, 2017, at 3:11 AM, Apache Jenkins Server <[hidden email]> wrote:
>
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1352/
>
> 1 tests failed.
> FAILED:  TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml.[empty]
>
> Error Message:
>
>
> Stack Trace:
> Test report file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml was length 0
>
>
>
> Build Log:
> [...truncated 262 lines...]
>   [junit4] JVM J0: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163200394456052706494.syserr
>   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
>   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>   [junit4] at java.security.AccessController.doPrivileged(Native Method)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
>   [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>   [junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>   [junit4] at java.io.PrintStream.flush(PrintStream.java:338)
>   [junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
>   [junit4] at java.io.PrintStream.write(PrintStream.java:482)
>   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>   [junit4] at java.io.Writer.write(Writer.java:157)
>   [junit4] at java.util.logging.StreamHandler.publish(StreamHandler.java:224)
>   [junit4] at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
>   [junit4] at java.util.logging.Logger.log(Logger.java:738)
>   [junit4] at java.util.logging.Logger.doLog(Logger.java:765)
>   [junit4] at java.util.logging.Logger.log(Logger.java:876)
>   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException(RandomizedRunner.java:531)
>   [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
>   [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
>   [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
>   [junit4] at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:28)
>   [junit4] at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
>   [junit4] Caused by: java.io.IOException: No space left on device
>   [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>   [junit4] at java.io.FileOut
>   [junit4] putStream.write(FileOutputStream.java:326)
>   [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>   [junit4] ... 26 more
>   [junit4] <<< JVM J0: EOF ----
>
> [...truncated 4 lines...]
>   [junit4] JVM J1: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6343353887351313582581.syserr
>   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
>   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>   [junit4] at java.security.AccessController.doPrivileged(Native Method)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testRunStarted(RunListenerEmitter.java:40)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testRunStarted(NoExceptionRunListenerDecorator.java:23)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testRunStarted(BeforeAfterRunListenerDecorator.java:24)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$1.notifyListener(OrderedRunNotifier.java:82)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:49)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestRunStarted(OrderedRunNotifier.java:79)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:249)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
>   [junit4] Caused by: java.io.IOException: No space left on device
>   [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>   [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
>   [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>   [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:121)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>   [junit4] at sun.nio.cs.Strea
>   [junit4] mEncoder.write(StreamEncoder.java:135)
>   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:24)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:31)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeDescription(AbstractEvent.java:119)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEventWithDescription.serialize(AbstractEventWithDescription.java:29)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.SuiteStartedEvent.serialize(SuiteStartedEvent.java:36)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>   [junit4] ... 13 more
>   [junit4] <<< JVM J1: EOF ----
>
>   [junit4] JVM J2: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345429632897914706142.syserr
>   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
>   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>   [junit4] at java.security.AccessController.doPrivileged(Native Method)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testFailure(RunListenerEmitter.java:54)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testFailure(NoExceptionRunListenerDecorator.java:55)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testFailure(BeforeAfterRunListenerDecorator.java:60)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$4.notifyListener(OrderedRunNotifier.java:129)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestFailure(OrderedRunNotifier.java:126)
>   [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$SubNotifier.fireTestFailure(ThreadLeakControl.java:184)
>   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:834)
>   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:922)
>   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>   [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>   [junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverr
>   [junit4] idesOnMethodsRule.java:40)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>   [junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>   [junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>   [junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>   [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>   [junit4] at java.lang.Thread.run(Thread.java:748)
>   [junit4] Caused by: java.io.IOException: No space left on device
>   [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>   [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
>   [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>   [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>   [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>   [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>   [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:566)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.FailureEvent.serialize(FailureEvent.java:46)
>   [junit4] at com.carrotsearch.ant.tasks.junit4
>   [junit4] .events.Serializer$2.run(Serializer.java:119)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>   [junit4] ... 30 more
>   [junit4] <<< JVM J2: EOF ----
>
> [...truncated 2 lines...]
>   [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163348084582467318555.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6166850120803246205.suites -stdin
>   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   [junit4] at java.lang.Thread.run(Thread.java:748)
>   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6341196093693912703591.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6348978994367824159899.suites -stdin
>   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   [junit4] at java.lang.Thread.run(Thread.java:748)
>   [junit4] ERROR: JVM J2 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6343973825201621029068.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345414770058120710840.suites -stdin
>   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   [junit4] at java.lang.Thread.run(Thread.java:748)
>
> BUILD FAILED
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:754: The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build.xml:50: The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1455: The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:985: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>
> Total time: 2 minutes 1 second
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> No artifacts from Lucene-Solr-NightlyTests-master #1349 to compare, so performing full copy of artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1352 - Still Failing

sarowe
I created <https://issues.apache.org/jira/browse/INFRA-14572> for the "lucene2” disk space problem.

On Hipchat Chris Thistlthwaite (@christ) said he would start looking at the box.

--
Steve
www.lucidworks.com

> On Jul 12, 2017, at 8:54 AM, Steve Rowe <[hidden email]> wrote:
>
> Looks like the new lucene2 node has run out of disk space.  I’ll go ask about it on Infra’s hipchat channel.
>
> --
> Steve
> www.lucidworks.com
>
>> On Jul 12, 2017, at 3:11 AM, Apache Jenkins Server <[hidden email]> wrote:
>>
>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1352/
>>
>> 1 tests failed.
>> FAILED:  TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml.[empty]
>>
>> Error Message:
>>
>>
>> Stack Trace:
>> Test report file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml was length 0
>>
>>
>>
>> Build Log:
>> [...truncated 262 lines...]
>>  [junit4] JVM J0: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163200394456052706494.syserr
>>  [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
>>  [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>>  [junit4] at java.security.AccessController.doPrivileged(Native Method)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
>>  [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>  [junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>  [junit4] at java.io.PrintStream.flush(PrintStream.java:338)
>>  [junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
>>  [junit4] at java.io.PrintStream.write(PrintStream.java:482)
>>  [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>  [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>>  [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>  [junit4] at java.io.Writer.write(Writer.java:157)
>>  [junit4] at java.util.logging.StreamHandler.publish(StreamHandler.java:224)
>>  [junit4] at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
>>  [junit4] at java.util.logging.Logger.log(Logger.java:738)
>>  [junit4] at java.util.logging.Logger.doLog(Logger.java:765)
>>  [junit4] at java.util.logging.Logger.log(Logger.java:876)
>>  [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException(RandomizedRunner.java:531)
>>  [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
>>  [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
>>  [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
>>  [junit4] at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:28)
>>  [junit4] at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
>>  [junit4] Caused by: java.io.IOException: No space left on device
>>  [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>>  [junit4] at java.io.FileOut
>>  [junit4] putStream.write(FileOutputStream.java:326)
>>  [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>>  [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>  [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>>  [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>>  [junit4] ... 26 more
>>  [junit4] <<< JVM J0: EOF ----
>>
>> [...truncated 4 lines...]
>>  [junit4] JVM J1: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6343353887351313582581.syserr
>>  [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
>>  [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>>  [junit4] at java.security.AccessController.doPrivileged(Native Method)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testRunStarted(RunListenerEmitter.java:40)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testRunStarted(NoExceptionRunListenerDecorator.java:23)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testRunStarted(BeforeAfterRunListenerDecorator.java:24)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$1.notifyListener(OrderedRunNotifier.java:82)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:49)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestRunStarted(OrderedRunNotifier.java:79)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:249)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
>>  [junit4] Caused by: java.io.IOException: No space left on device
>>  [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>>  [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>  [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>  [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:121)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>>  [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>  [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>  [junit4] at sun.nio.cs.Strea
>>  [junit4] mEncoder.write(StreamEncoder.java:135)
>>  [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:24)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:31)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeDescription(AbstractEvent.java:119)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEventWithDescription.serialize(AbstractEventWithDescription.java:29)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.SuiteStartedEvent.serialize(SuiteStartedEvent.java:36)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>>  [junit4] ... 13 more
>>  [junit4] <<< JVM J1: EOF ----
>>
>>  [junit4] JVM J2: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345429632897914706142.syserr
>>  [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
>>  [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>>  [junit4] at java.security.AccessController.doPrivileged(Native Method)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testFailure(RunListenerEmitter.java:54)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testFailure(NoExceptionRunListenerDecorator.java:55)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testFailure(BeforeAfterRunListenerDecorator.java:60)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$4.notifyListener(OrderedRunNotifier.java:129)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestFailure(OrderedRunNotifier.java:126)
>>  [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$SubNotifier.fireTestFailure(ThreadLeakControl.java:184)
>>  [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:834)
>>  [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:922)
>>  [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>>  [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>>  [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>  [junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>  [junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverr
>>  [junit4] idesOnMethodsRule.java:40)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>  [junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>  [junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>  [junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>  [junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>  [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>  [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>  [junit4] at java.lang.Thread.run(Thread.java:748)
>>  [junit4] Caused by: java.io.IOException: No space left on device
>>  [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>>  [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>  [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>>  [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>  [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>  [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>>  [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:566)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.FailureEvent.serialize(FailureEvent.java:46)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4
>>  [junit4] .events.Serializer$2.run(Serializer.java:119)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>>  [junit4] ... 30 more
>>  [junit4] <<< JVM J2: EOF ----
>>
>> [...truncated 2 lines...]
>>  [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163348084582467318555.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6166850120803246205.suites -stdin
>>  [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>>  [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>  [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>  [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>  [junit4] at java.lang.Thread.run(Thread.java:748)
>>  [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6341196093693912703591.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6348978994367824159899.suites -stdin
>>  [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>>  [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>  [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>  [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>  [junit4] at java.lang.Thread.run(Thread.java:748)
>>  [junit4] ERROR: JVM J2 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6343973825201621029068.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345414770058120710840.suites -stdin
>>  [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>>  [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>>  [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>  [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>  [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>  [junit4] at java.lang.Thread.run(Thread.java:748)
>>
>> BUILD FAILED
>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:754: The following error occurred while executing this line:
>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build.xml:50: The following error occurred while executing this line:
>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1455: The following error occurred while executing this line:
>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:985: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>
>> Total time: 2 minutes 1 second
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> No artifacts from Lucene-Solr-NightlyTests-master #1349 to compare, so performing full copy of artifacts
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1352 - Still Failing

sarowe
The disk allocation for the “lucene2” node was increased and it’s back in rotation now after being offline for a while to perform the operation.  <https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.0/3/> is in progress now.

--
Steve
www.lucidworks.com

> On Jul 12, 2017, at 9:26 AM, Steve Rowe <[hidden email]> wrote:
>
> I created <https://issues.apache.org/jira/browse/INFRA-14572> for the "lucene2” disk space problem.
>
> On Hipchat Chris Thistlthwaite (@christ) said he would start looking at the box.
>
> --
> Steve
> www.lucidworks.com
>
>> On Jul 12, 2017, at 8:54 AM, Steve Rowe <[hidden email]> wrote:
>>
>> Looks like the new lucene2 node has run out of disk space.  I’ll go ask about it on Infra’s hipchat channel.
>>
>> --
>> Steve
>> www.lucidworks.com
>>
>>> On Jul 12, 2017, at 3:11 AM, Apache Jenkins Server <[hidden email]> wrote:
>>>
>>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1352/
>>>
>>> 1 tests failed.
>>> FAILED:  TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml.[empty]
>>>
>>> Error Message:
>>>
>>>
>>> Stack Trace:
>>> Test report file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/TEST-org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.xml was length 0
>>>
>>>
>>>
>>> Build Log:
>>> [...truncated 262 lines...]
>>> [junit4] JVM J0: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163200394456052706494.syserr
>>> [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
>>> [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>>> [junit4] at java.security.AccessController.doPrivileged(Native Method)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
>>> [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> [junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>> [junit4] at java.io.PrintStream.flush(PrintStream.java:338)
>>> [junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
>>> [junit4] at java.io.PrintStream.write(PrintStream.java:482)
>>> [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>> [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>>> [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>> [junit4] at java.io.Writer.write(Writer.java:157)
>>> [junit4] at java.util.logging.StreamHandler.publish(StreamHandler.java:224)
>>> [junit4] at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:116)
>>> [junit4] at java.util.logging.Logger.log(Logger.java:738)
>>> [junit4] at java.util.logging.Logger.doLog(Logger.java:765)
>>> [junit4] at java.util.logging.Logger.log(Logger.java:876)
>>> [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException(RandomizedRunner.java:531)
>>> [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
>>> [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
>>> [junit4] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
>>> [junit4] at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:28)
>>> [junit4] at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
>>> [junit4] Caused by: java.io.IOException: No space left on device
>>> [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>>> [junit4] at java.io.FileOut
>>> [junit4] putStream.write(FileOutputStream.java:326)
>>> [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>>> [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>> [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>>> [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>>> [junit4] ... 26 more
>>> [junit4] <<< JVM J0: EOF ----
>>>
>>> [...truncated 4 lines...]
>>> [junit4] JVM J1: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6343353887351313582581.syserr
>>> [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
>>> [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>>> [junit4] at java.security.AccessController.doPrivileged(Native Method)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testRunStarted(RunListenerEmitter.java:40)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testRunStarted(NoExceptionRunListenerDecorator.java:23)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testRunStarted(BeforeAfterRunListenerDecorator.java:24)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$1.notifyListener(OrderedRunNotifier.java:82)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:49)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestRunStarted(OrderedRunNotifier.java:79)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:249)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
>>> [junit4] Caused by: java.io.IOException: No space left on device
>>> [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>>> [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>> [junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:121)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>>> [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>> [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>> [junit4] at sun.nio.cs.Strea
>>> [junit4] mEncoder.write(StreamEncoder.java:135)
>>> [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:24)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.JsonHelpers.writeDescription(JsonHelpers.java:31)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeDescription(AbstractEvent.java:119)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.AbstractEventWithDescription.serialize(AbstractEventWithDescription.java:29)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.SuiteStartedEvent.serialize(SuiteStartedEvent.java:36)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>>> [junit4] ... 13 more
>>> [junit4] <<< JVM J1: EOF ----
>>>
>>> [junit4] JVM J2: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345429632897914706142.syserr
>>> [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
>>> [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
>>> [junit4] at java.security.AccessController.doPrivileged(Native Method)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testFailure(RunListenerEmitter.java:54)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testFailure(NoExceptionRunListenerDecorator.java:55)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testFailure(BeforeAfterRunListenerDecorator.java:60)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$4.notifyListener(OrderedRunNotifier.java:129)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestFailure(OrderedRunNotifier.java:126)
>>> [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$SubNotifier.fireTestFailure(ThreadLeakControl.java:184)
>>> [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:834)
>>> [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:922)
>>> [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>>> [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>>> [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>> [junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> [junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverr
>>> [junit4] idesOnMethodsRule.java:40)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> [junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>> [junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>> [junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>> [junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>> [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>> [junit4] at java.lang.Thread.run(Thread.java:748)
>>> [junit4] Caused by: java.io.IOException: No space left on device
>>> [junit4] at java.io.FileOutputStream.writeBytes(Native Method)
>>> [junit4] at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>> [junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
>>> [junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>>> [junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
>>> [junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
>>> [junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:566)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.FailureEvent.serialize(FailureEvent.java:46)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4
>>> [junit4] .events.Serializer$2.run(Serializer.java:119)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
>>> [junit4] ... 30 more
>>> [junit4] <<< JVM J2: EOF ----
>>>
>>> [...truncated 2 lines...]
>>> [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6163348084582467318555.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170712_070941_6166850120803246205.suites -stdin
>>> [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>>> [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [junit4] at java.lang.Thread.run(Thread.java:748)
>>> [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6341196093693912703591.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20170712_070941_6348978994367824159899.suites -stdin
>>> [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>>> [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [junit4] at java.lang.Thread.run(Thread.java:748)
>>> [junit4] ERROR: JVM J2 ended with an exception, command line: /usr/local/asfpackages/java/jdk1.8.0_131/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=55E4911CB74D62C9 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp -Dcommon.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene -Dclover.db.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/clover/db -Djava.security.policy=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout -Djunit4.childvm.cwd=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/codecs/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/test-framework/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/test-framework/lib/randomizedtesting-runner-2.5.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/classes/test:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_131/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6343973825201621029068.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20170712_070941_6345414770058120710840.suites -stdin
>>> [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
>>> [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
>>> [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [junit4] at java.lang.Thread.run(Thread.java:748)
>>>
>>> BUILD FAILED
>>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:818: The following error occurred while executing this line:
>>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:754: The following error occurred while executing this line:
>>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
>>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build.xml:50: The following error occurred while executing this line:
>>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1455: The following error occurred while executing this line:
>>> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:985: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
>>>
>>> Total time: 2 minutes 1 second
>>> Build step 'Invoke Ant' marked build as failure
>>> Archiving artifacts
>>> No artifacts from Lucene-Solr-NightlyTests-master #1349 to compare, so performing full copy of artifacts
>>> Recording test results
>>> Email was triggered for: Failure - Any
>>> Sending email for trigger: Failure - Any
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1353 - Still unstable

Apache Jenkins Server-2
In reply to this post by Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1353/

4 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([35CE8052BB09C003:BD9ABF8815F5ADFB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=17124, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=17123, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   1) Thread[id=17124, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=17123, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
        at __randomizedtesting.SeedInfo.seed([35CE8052BB09C003]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=17124, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    2) Thread[id=17123, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.io.PrintStream.println(PrintStream.java:805)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=17124, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   2) Thread[id=17123, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
        at __randomizedtesting.SeedInfo.seed([35CE8052BB09C003]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1499932834675,generation=2,filelist=[_13k.cfe, _13k.cfs, _13k.si, _13l.fdt, _13l.fdx, _13l.fnm, _13l.nvd, _13l.nvm, _13l.si, _13l_Memory_0.ram, _13m.fdt, _13m.fdx, _13m.fnm, _13m.nvd, _13m.nvm, _13m.si, _13m_Memory_0.ram, _fo.fdt, _fo.fdx, _fo.fnm, _fo.nvd, _fo.nvm, _fo.si, _fo_Memory_0.ram, _v1.cfe, _v1.cfs, _v1.si, segments_2]}]> but was:<[{indexVersion=1499932834675,generation=2,filelist=[_13k.cfe, _13k.cfs, _13k.si, _13l.fdt, _13l.fdx, _13l.fnm, _13l.nvd, _13l.nvm, _13l.si, _13l_Memory_0.ram, _13m.fdt, _13m.fdx, _13m.fnm, _13m.nvd, _13m.nvm, _13m.si, _13m_Memory_0.ram, _fo.fdt, _fo.fdx, _fo.fnm, _fo.nvd, _fo.nvm, _fo.si, _fo_Memory_0.ram, _v1.cfe, _v1.cfs, _v1.si, segments_2]}, {indexVersion=1499932834675,generation=3,filelist=[_13n.cfe, _13n.cfs, _13n.si, _fo.fdt, _fo.fdx, _fo.fnm, _fo.nvd, _fo.nvm, _fo.si, _fo_Memory_0.ram, _v1.cfe, _v1.cfs, _v1.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1499932834675,generation=2,filelist=[_13k.cfe, _13k.cfs, _13k.si, _13l.fdt, _13l.fdx, _13l.fnm, _13l.nvd, _13l.nvm, _13l.si, _13l_Memory_0.ram, _13m.fdt, _13m.fdx, _13m.fnm, _13m.nvd, _13m.nvm, _13m.si, _13m_Memory_0.ram, _fo.fdt, _fo.fdx, _fo.fnm, _fo.nvd, _fo.nvm, _fo.si, _fo_Memory_0.ram, _v1.cfe, _v1.cfs, _v1.si, segments_2]}]> but was:<[{indexVersion=1499932834675,generation=2,filelist=[_13k.cfe, _13k.cfs, _13k.si, _13l.fdt, _13l.fdx, _13l.fnm, _13l.nvd, _13l.nvm, _13l.si, _13l_Memory_0.ram, _13m.fdt, _13m.fdx, _13m.fnm, _13m.nvd, _13m.nvm, _13m.si, _13m_Memory_0.ram, _fo.fdt, _fo.fdx, _fo.fnm, _fo.nvd, _fo.nvm, _fo.si, _fo_Memory_0.ram, _v1.cfe, _v1.cfs, _v1.si, segments_2]}, {indexVersion=1499932834675,generation=3,filelist=[_13n.cfe, _13n.cfs, _13n.si, _fo.fdt, _fo.fdx, _fo.fnm, _fo.nvd, _fo.nvm, _fo.si, _fo_Memory_0.ram, _v1.cfe, _v1.cfs, _v1.si, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([35CE8052BB09C003:10199B62CB41CE00]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11960 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_35CE8052BB09C003-001/init-core-data-001
   [junit4]   2> 2504486 WARN  (SUITE-ChaosMonkeySafeLeaderTest-seed#[35CE8052BB09C003]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 2504487 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[35CE8052BB09C003]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2504500 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[35CE8052BB09C003]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2504500 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[35CE8052BB09C003]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2504502 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2504502 INFO  (Thread-10065) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2504502 INFO  (Thread-10065) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2504503 ERROR (Thread-10065) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2504605 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.ZkTestServer start zk server on port:32847
   [junit4]   2> 2504644 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2504645 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2504663 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2504664 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2504665 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2504665 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2504666 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2504679 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2504680 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2504680 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2504681 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2504682 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2505188 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2505224 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2f15d3b{/,null,AVAILABLE}
   [junit4]   2> 2505224 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@147b884d{HTTP/1.1,[http/1.1]}{127.0.0.1:45895}
   [junit4]   2> 2505224 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.e.j.s.Server Started @2514681ms
   [junit4]   2> 2505224 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_35CE8052BB09C003-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=45895, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_35CE8052BB09C003-001/control-001/cores}
   [junit4]   2> 2505224 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2505225 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2505225 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2505225 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 2505225 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-13T07:52:47.774Z
   [junit4]   2> 2505237 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2505237 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_35CE8052BB09C003-001/control-001/solr.xml
   [junit4]   2> 2505262 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7b043e21, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2505264 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32847/solr
   [junit4]   2> 2505801 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2505801 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45895_
   [junit4]   2> 2505802 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.c.Overseer Overseer (id=98299567607119876-127.0.0.1:45895_-n_0000000000) starting
   [junit4]   2> 2505876 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45895_
   [junit4]   2> 2505877 INFO  (zkCallback-1064-thread-1-processing-n:127.0.0.1:45895_) [n:127.0.0.1:45895_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2506342 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b043e21
   [junit4]   2> 2506345 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b043e21
   [junit4]   2> 2506398 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b043e21
   [junit4]   2> 2506399 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [n:127.0.0.1:45895_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_35CE8052BB09C003-001/control-001/cores
   [junit4]   2> 2506597 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2506670 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[35CE8052BB09C003]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32847/solr ready
   [junit4]   2> 2506707 INFO  (qtp261395121-16743) [n:127.0.0.1:45895_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:45895_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2506764 INFO  (OverseerThreadFactory-2644-thread-1-processing-n:127.0.0.1:45895_) [n:127.0.0.1:45895_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2507073 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2507073 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2507182 INFO  (zkCallback-1064-thread-1-processing-n:127.0.0.1:45895_) [n:127.0.0.1:45895_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2508161 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2508248 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2508681 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2508734 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2508735 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b043e21
   [junit4]   2> 2508735 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2508735 INFO  (qtp261395121-16741) [n:127.0.0.1:45895_ c:control_collection s:shard1 &nb