[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1295 - Failure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1295 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1295/

7 tests failed.
FAILED:  org.apache.solr.TestGroupingSearch.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
        at java.lang.String.<init>(String.java:207)
        at java.lang.StringBuilder.toString(StringBuilder.java:407)
        at java.lang.reflect.Executable.sharedToGenericString(Executable.java:187)
        at java.lang.reflect.Method.toGenericString(Method.java:415)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:26)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
        at java.util.TimSort.binarySort(TimSort.java:296)
        at java.util.TimSort.sort(TimSort.java:239)
        at java.util.Arrays.sort(Arrays.java:1438)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
        at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.<init>(RandomizedRunner.java:329)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:31)
        at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:24)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:239)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at java.lang.reflect.Executable.sharedToGenericString(Executable.java:171)
        at java.lang.reflect.Method.toGenericString(Method.java:415)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:26)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
        at java.util.TimSort.binarySort(TimSort.java:296)
        at java.util.TimSort.sort(TimSort.java:239)
        at java.util.Arrays.sort(Arrays.java:1438)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
        at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.<init>(RandomizedRunner.java:329)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:31)
        at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:24)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:239)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)


FAILED:  org.apache.solr.handler.admin.ShowFileRequestHandlerTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.reflect.Modifier.toString(Modifier.java:235)
        at java.lang.reflect.Executable.printModifiersIfNonzero(Executable.java:100)
        at java.lang.reflect.Executable.sharedToGenericString(Executable.java:146)
        at java.lang.reflect.Method.toGenericString(Method.java:415)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:26)
        at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
        at java.util.TimSort.binarySort(TimSort.java:296)
        at java.util.TimSort.sort(TimSort.java:239)
        at java.util.Arrays.sort(Arrays.java:1438)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
        at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
        at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:212)
        at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.<init>(RandomizedRunner.java:329)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:31)
        at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:24)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
        at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:239)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:45652/wxiz/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:45652/wxiz/collection1
        at __randomizedtesting.SeedInfo.seed([E76CDD475AEFCD4D:6F38E29DF413A0B5]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:638)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:451)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:392)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1535)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:391)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:160)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:532)
        ... 54 more


FAILED:  junit.framework.TestSuite.org.apache.solr.update.TestInPlaceUpdatesDistrib

Error Message:
7 threads leaked from SUITE scope at org.apache.solr.update.TestInPlaceUpdatesDistrib:     1) Thread[id=134, name=qtp1986118145-134, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]         at java.util.HashMap.resize(HashMap.java:734)         at java.util.HashMap.putVal(HashMap.java:662)         at java.util.HashMap.put(HashMap.java:611)         at java.util.HashSet.add(HashSet.java:219)         at java.util.AbstractCollection.addAll(AbstractCollection.java:344)         at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:792)         at org.apache.lucene.index.IndexFileDeleter$CommitPoint.<init>(IndexFileDeleter.java:789)         at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:524)         at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3297)         at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3254)         at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3209)         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:682)         at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)         at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)         at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)         at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=97, name=searcherExecutor-31-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=162, name=searcherExecutor-53-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=76, name=qtp1174328245-76, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]         at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507)         at org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:257)         at org.apache.lucene.index.BufferedUpdatesStream.applyDocValuesUpdates(BufferedUpdatesStream.java:667)         at org.apache.lucene.index.BufferedUpdatesStream.applyDocValuesUpdatesList(BufferedUpdatesStream.java:613)         at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:269)         at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)         at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3421)         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3378)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1125)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1170)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:280)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:898)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:96)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:392)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:81)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1539)         at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:581)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=101, name=qtp1013664760-101, state=BLOCKED, group=TGRP-TestInPlaceUpdatesDistrib]         at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4867)         at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)         at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:278)         at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235)         at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1990)         at org.apache.solr.update.UpdateLog.openRealtimeSearcher(UpdateLog.java:638)         at org.apache.solr.update.IndexFingerprint.getFingerprint(IndexFingerprint.java:94)         at org.apache.solr.handler.component.RealTimeGetComponent.processGetFingeprint(RealTimeGetComponent.java:947)         at org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:123)         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=106, name=qtp1013664760-106, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]         at java.util.HashSet.<init>(HashSet.java:118)         at org.apache.lucene.index.SegmentCommitInfo.files(SegmentCommitInfo.java:220)         at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:792)         at org.apache.lucene.index.IndexFileDeleter.incRef(IndexFileDeleter.java:553)         at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:520)         at org.apache.lucene.index.IndexWriter.checkpoint(IndexWriter.java:2481)         at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:651)         at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)         at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)         at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)         at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3050)         at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3246)         at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3209)         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:682)         at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)         at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)         at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)         at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=125, name=searcherExecutor-42-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 7 threads leaked from SUITE scope at org.apache.solr.update.TestInPlaceUpdatesDistrib:
   1) Thread[id=134, name=qtp1986118145-134, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]
        at java.util.HashMap.resize(HashMap.java:734)
        at java.util.HashMap.putVal(HashMap.java:662)
        at java.util.HashMap.put(HashMap.java:611)
        at java.util.HashSet.add(HashSet.java:219)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
        at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:792)
        at org.apache.lucene.index.IndexFileDeleter$CommitPoint.<init>(IndexFileDeleter.java:789)
        at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:524)
        at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3297)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3254)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3209)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:682)
        at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
        at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=97, name=searcherExecutor-31-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=162, name=searcherExecutor-53-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=76, name=qtp1174328245-76, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]
        at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507)
        at org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.seekExact(AssertingLeafReader.java:257)
        at org.apache.lucene.index.BufferedUpdatesStream.applyDocValuesUpdates(BufferedUpdatesStream.java:667)
        at org.apache.lucene.index.BufferedUpdatesStream.applyDocValuesUpdatesList(BufferedUpdatesStream.java:613)
        at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:269)
        at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)
        at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3421)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3378)
        at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1125)
        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1170)
        at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:280)
        at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:898)
        at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:96)
        at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:392)
        at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:81)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1539)
        at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:581)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=101, name=qtp1013664760-101, state=BLOCKED, group=TGRP-TestInPlaceUpdatesDistrib]
        at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4867)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:278)
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1990)
        at org.apache.solr.update.UpdateLog.openRealtimeSearcher(UpdateLog.java:638)
        at org.apache.solr.update.IndexFingerprint.getFingerprint(IndexFingerprint.java:94)
        at org.apache.solr.handler.component.RealTimeGetComponent.processGetFingeprint(RealTimeGetComponent.java:947)
        at org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:123)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=106, name=qtp1013664760-106, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]
        at java.util.HashSet.<init>(HashSet.java:118)
        at org.apache.lucene.index.SegmentCommitInfo.files(SegmentCommitInfo.java:220)
        at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:792)
        at org.apache.lucene.index.IndexFileDeleter.incRef(IndexFileDeleter.java:553)
        at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:520)
        at org.apache.lucene.index.IndexWriter.checkpoint(IndexWriter.java:2481)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:651)
        at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)
        at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)
        at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3050)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3246)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3209)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:682)
        at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
        at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=125, name=searcherExecutor-42-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([E76CDD475AEFCD4D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.TestInPlaceUpdatesDistrib

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=134, name=qtp1986118145-134, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]         at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:714)         at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)         at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)         at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)         at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)         at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:278)         at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235)         at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1990)         at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2199)         at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1936)         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:713)         at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)         at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)         at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)         at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=97, name=searcherExecutor-31-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=162, name=searcherExecutor-53-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=76, name=qtp1174328245-76, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]         at java.util.HashMap.newNode(HashMap.java:1742)         at java.util.HashMap.putVal(HashMap.java:641)         at java.util.HashMap.put(HashMap.java:611)         at java.util.HashSet.add(HashSet.java:219)         at java.util.AbstractCollection.addAll(AbstractCollection.java:344)         at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:792)         at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:540)         at org.apache.lucene.index.IndexWriter.checkpoint(IndexWriter.java:2481)         at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:651)         at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)         at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)         at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3421)         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3378)         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1125)         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1170)         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:280)         at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:898)         at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:96)         at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:392)         at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:81)         at org.apache.solr.core.SolrCore.close(SolrCore.java:1539)         at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:581)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=101, name=qtp1013664760-101, state=BLOCKED, group=TGRP-TestInPlaceUpdatesDistrib]         at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4867)         at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)         at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:278)         at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235)         at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1990)         at org.apache.solr.update.UpdateLog.openRealtimeSearcher(UpdateLog.java:638)         at org.apache.solr.update.IndexFingerprint.getFingerprint(IndexFingerprint.java:94)         at org.apache.solr.handler.component.RealTimeGetComponent.processGetFingeprint(RealTimeGetComponent.java:947)         at org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:123)         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=106, name=qtp1013664760-106, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]         at java.lang.Thread.holdsLock(Native Method)         at org.apache.lucene.index.IndexFileDeleter.locked(IndexFileDeleter.java:117)         at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:630)         at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:583)         at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:534)         at org.apache.lucene.index.IndexWriter.checkpoint(IndexWriter.java:2481)         at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:651)         at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)         at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)         at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)         at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3050)         at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3246)         at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3209)         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:682)         at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)         at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)         at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)         at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)         at org.eclipse.jetty.server.Server.handle(Server.java:534)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=125, name=searcherExecutor-42-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=134, name=qtp1986118145-134, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]
        at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:714)
        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:467)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:278)
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1990)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2199)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1936)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:713)
        at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1937)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1913)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
        at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:62)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:178)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2487)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:722)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:528)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=97, name=searcherExecutor-31-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=162, name=searcherExecutor-53-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=76, name=qtp1174328245-76, state=RUNNABLE, group=TGRP-TestInPlaceUpdatesDistrib]
        at java.util.HashMap.newNode(HashMap.java:1742)
        at java.util.HashMap.putVal(HashMap.java:641)
        at java.util.HashMap.put(HashMap.java:611)
        at java.util.HashSet.add(HashSet.java:219)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
        at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:792)
        at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:540)
        at org.apache.lucene.index.IndexWriter.checkpoint(IndexWriter.java:2481)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:651)
        at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)
        at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3458)
        at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3444)
        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3421)
        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3378)
        at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1125)
        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1170)
        at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:280)
        at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:898)
        at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:96)
        at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:392)
        at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:81)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1539)
        at org.apache.solr.servlet.HttpSolrCall.destroy(HttpSolrCall.java:581)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:312)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=101, name=qtp1013664760-101, state=BLOCKED, group=TGRP-TestInPlaceUpdatesDistrib]
        at org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4867)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:278)
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:235)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1990)
        at org.apache.solr.update.UpdateLog.openRealtimeSe

[...truncated too long message...]

t org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    >         at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]    >         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    >         at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    >         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    >         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    >         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
   [junit4]    >         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    >         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    >         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    7) Thread[id=125, name=searcherExecutor-42-thread-1, state=WAITING, group=TGRP-TestInPlaceUpdatesDistrib]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([E76CDD475AEFCD4D]:0)
   [junit4] Completed [375/709 (2!)] on J2 in 3693.38s, 1 test, 3 errors <<< FAILURES!

[...truncated 49 lines...]
   [junit4] Suite: org.apache.solr.TestGroupingSearch
   [junit4] ERROR   0.00s J2 | TestGroupingSearch.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > at java.util.Arrays.copyOfRange(Arrays.java:3664)
   [junit4]    > at java.lang.String.<init>(String.java:207)
   [junit4]    > at java.lang.StringBuilder.toString(StringBuilder.java:407)
   [junit4]    > at java.lang.reflect.Executable.sharedToGenericString(Executable.java:187)
   [junit4]    > at java.lang.reflect.Method.toGenericString(Method.java:415)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:26)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
   [junit4]    > at java.util.TimSort.binarySort(TimSort.java:296)
   [junit4]    > at java.util.TimSort.sort(TimSort.java:239)
   [junit4]    > at java.util.Arrays.sort(Arrays.java:1438)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
   [junit4]    > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4] Completed [392/709 (3!)] on J2 in 0.38s, 1 test, 1 error <<< FAILURES!

[...truncated 4 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.ShowFileRequestHandlerTest
   [junit4] ERROR   0.50s J2 | ShowFileRequestHandlerTest.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > at java.lang.reflect.Modifier.toString(Modifier.java:235)
   [junit4]    > at java.lang.reflect.Executable.printModifiersIfNonzero(Executable.java:100)
   [junit4]    > at java.lang.reflect.Executable.sharedToGenericString(Executable.java:146)
   [junit4]    > at java.lang.reflect.Method.toGenericString(Method.java:415)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:26)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
   [junit4]    > at java.util.TimSort.binarySort(TimSort.java:296)
   [junit4]    > at java.util.TimSort.sort(TimSort.java:239)
   [junit4]    > at java.util.Arrays.sort(Arrays.java:1438)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
   [junit4]    > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4] Completed [394/709 (4!)] on J2 in 0.50s, 1 test, 1 error <<< FAILURES!

[...truncated 4 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSegmentSorting
   [junit4] ERROR   0.00s J2 | TestSegmentSorting.initializationError <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > at java.util.Arrays.copyOf(Arrays.java:3332)
   [junit4]    > at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
   [junit4]    > at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
   [junit4]    > at java.lang.StringBuilder.append(StringBuilder.java:136)
   [junit4]    > at java.lang.reflect.Executable.sharedToGenericString(Executable.java:171)
   [junit4]    > at java.lang.reflect.Method.toGenericString(Method.java:415)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:26)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$1.compare(ClassModel.java:23)
   [junit4]    > at java.util.TimSort.binarySort(TimSort.java:296)
   [junit4]    > at java.util.TimSort.sort(TimSort.java:239)
   [junit4]    > at java.util.Arrays.sort(Arrays.java:1438)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:216)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$3.members(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel$ModelBuilder.build(ClassModel.java:85)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.methodsModel(ClassModel.java:212)
   [junit4]    > at com.carrotsearch.randomizedtesting.ClassModel.<init>(ClassModel.java:207)
   [junit4]    > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4] Completed [396/709 (5!)] on J2 in 0.00s, 1 test, 1 error <<< FAILURES!

[...truncated 970 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20170505_113721_5573700493991963119845.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid361.hprof ...
   [junit4] Heap dump file created [669120947 bytes in 4.178 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 7457 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:727: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid361.hprof

Total time: 269 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1296 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1296/

5 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([9A26FFB17569DD8E]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexSorting

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([9A26FFB17569DD8E]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest.test

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
        at __randomizedtesting.SeedInfo.seed([D075F912027B4DA5:5821C6C8AC87205D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:187)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:855)
        at org.apache.solr.cloud.TlogReplayBufferedWhileIndexingTest.test(TlogReplayBufferedWhileIndexingTest.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:59623/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:57502/solr/test_col_shard3_replica1: Server Error    request: http://127.0.0.1:57502/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A59623%2Fsolr%2Ftest_col_shard5_replica2%2F&wt=javabin&version=2 Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42815/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@19b8638c

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:59623/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:57502/solr/test_col_shard3_replica1: Server Error



request: http://127.0.0.1:57502/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A59623%2Fsolr%2Ftest_col_shard5_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42815/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@19b8638c
        at __randomizedtesting.SeedInfo.seed([D075F912027B4DA5:E6619B54882677B4]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:59623/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:57502/solr/test_col_shard3_replica1: Server Error



request: http://127.0.0.1:57502/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A59623%2Fsolr%2Ftest_col_shard5_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42815/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@19b8638c
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:59623/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:57502/solr/test_col_shard3_replica1: Server Error



request: http://127.0.0.1:57502/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A59623%2Fsolr%2Ftest_col_shard5_replica2%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:42815/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@19b8638c
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
        ... 6 more


FAILED:  org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([D075F912027B4DA5:28AB6CFA7A179CF6]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:896)
        at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion(SegmentsInfoRequestHandlerTest.java:70)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=2=count(//lst[@name='segments']/lst/str[@name='version'][.='7.0.0'])
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">51</int></lst><lst name="segments"><lst name="_0"><str name="name">_0</str><int name="delCount">1</int><long name="sizeInBytes">2168</long><int name="size">4</int><date name="age">2017-05-10T07:46:54.661Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_1"><str name="name">_1</str><int name="delCount">0</int><long name="sizeInBytes">1990</long><int name="size">1</int><date name="age">2017-05-10T07:46:54.663Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_2"><str name="name">_2</str><int name="delCount">0</int><long name="sizeInBytes">2101</long><int name="size">4</int><date name="age">2017-05-10T07:46:54.672Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_3"><str name="name">_3</str><int name="delCount">0</int><long name="sizeInBytes">1990</long><int name="size">1</int><date name="age">2017-05-10T07:46:54.678Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst></lst>
</response>

        request was:qt=/admin/segments&wt=xml
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:889)
        ... 40 more




Build Log:
[...truncated 1729 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> mai 10, 2017 3:48:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexSorting
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    2) Thread[id=1621, name=TEST-TestIndexSorting.testRandom3-seed#[9A26FFB17569DD8E], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:981)
   [junit4]   2>         at org.apache.lucene.util.fst.FST.readFirstRealTargetArc(FST.java:883)
   [junit4]   2>         at org.apache.lucene.util.fst.NodeHash.hash(NodeHash.java:98)
   [junit4]   2>         at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:148)
   [junit4]   2>         at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   [junit4]   2>         at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   [junit4]   2>         at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]   2>         at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
   [junit4]   2>         at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
   [junit4]   2>         at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:373)
   [junit4]   2>         at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:416)
   [junit4]   2>         at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:406)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:527)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:139)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4361)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3933)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2082)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4992)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5030)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5021)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1573)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1315)
   [junit4]   2>         at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2300)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    3) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    4) Thread[id=1620, name=SUITE-TestIndexSorting-seed#[9A26FFB17569DD8E], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1556)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.method=testRandom3 -Dtests.seed=9A26FFB17569DD8E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=nn-NO -Dtests.timezone=SST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7180s J1 | TestIndexSorting.testRandom3 <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9A26FFB17569DD8E]:0)
   [junit4]   2> mai 10, 2017 3:48:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> mai 10, 2017 3:48:49 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexSorting:
   [junit4]   2>    1) Thread[id=1621, name=TEST-TestIndexSorting.testRandom3-seed#[9A26FFB17569DD8E], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:133)
   [junit4]   2>         at org.apache.lucene.util.ThrottledIndexOutput.writeBytes(ThrottledIndexOutput.java:98)
   [junit4]   2>         at org.apache.lucene.util.ThrottledIndexOutput.writeByte(ThrottledIndexOutput.java:89)
   [junit4]   2>         at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:73)
   [junit4]   2>         at org.apache.lucene.store.DataOutput.writeLong(DataOutput.java:213)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericFieldValues(DirectDocValuesConsumer.java:138)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericField(DirectDocValuesConsumer.java:73)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:175)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4361)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3933)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2082)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4992)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5030)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5021)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1573)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1315)
   [junit4]   2>         at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2300)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> mai 10, 2017 3:48:49 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=1621, name=TEST-TestIndexSorting.testRandom3-seed#[9A26FFB17569DD8E], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J1/temp/lucene.index.TestIndexSorting_9A26FFB17569DD8E-001
   [junit4]   2> May 10, 2017 4:48:49 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {sparse_text=FST50, id=PostingsFormat(name=Asserting)}, docValues:{sparse_text=DocValuesFormat(name=Lucene70), multi_valued_long=DocValuesFormat(name=Lucene70), double=DocValuesFormat(name=Lucene70), foo=DocValuesFormat(name=Lucene70), sparse_binary=DocValuesFormat(name=Lucene70), float=DocValuesFormat(name=Direct), int=DocValuesFormat(name=Memory), long=DocValuesFormat(name=Direct), sparse_int=DocValuesFormat(name=Direct), multi_valued_double=DocValuesFormat(name=Memory), sparse=DocValuesFormat(name=Lucene70), bytes=DocValuesFormat(name=Memory), dense_int=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Memory), multi_valued_int=DocValuesFormat(name=Lucene70), multi_valued_bytes=DocValuesFormat(name=Lucene70), multi_valued_float=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=442, maxMBSortInHeap=6.929891466375805, sim=RandomSimilarity(queryNorm=true): {id=DFI(ChiSquared)}, locale=nn-NO, timezone=SST
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=169321472,total=495976448
   [junit4]   2> NOTE: All tests run in this JVM: [TestPrefixInBooleanQuery, TestBooleanOr, TestTimSorter, TestWeakIdentityMap, TestBoostQuery, TestIntroSorter, TestTermQuery, TestSizeBoundedForceMerge, TestTermsEnum2, TestIndexWriterOutOfFileDescriptors, TestDocValuesQueries, TestRegexpRandom2, TestSimpleExplanationsWithFillerDocs, TestPrefixCodedTerms, TestDocIdsWriter, TestRateLimiter, TestOmitPositions, TestPrefixRandom, TestDirectory, Test2BDocs, TestDocValues, TestBufferedChecksum, TestPriorityQueue, TestDocumentsWriterDeleteQueue, TestDeterminism, TestAllFilesDetectTruncation, TestNorms, TestIndexWriterOnJRECrash, TestSparseFixedBitSet, TestDemo, TestWildcard, TestSortedSetSelector, TestSearcherManager, TestInPlaceMergeSorter, TestSpanOrQuery, TestMmapDirectory, TestDuelingCodecs, TestTragicIndexWriterDeadlock, TestDoc, TestMergeSchedulerExternal, TestSumDocFreq, TestOmitTf, TestByteArrayDataInput, TestIndexSorting]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.seed=9A26FFB17569DD8E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=nn-NO -Dtests.timezone=SST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestIndexSorting (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9A26FFB17569DD8E]:0)
   [junit4] Completed [420/451 (1!)] on J1 in 7220.35s, 42 tests, 2 errors <<< FAILURES!

[...truncated 10542 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/init-core-data-001
   [junit4]   2> 1010086 WARN  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
   [junit4]   2> 1010086 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1010088 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=SSL overhead seems to cause OutOfMemory when stress testing)
   [junit4]   2> 1010089 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 10 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001
   [junit4]   2> 1010089 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1010090 INFO  (Thread-4333) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1010090 INFO  (Thread-4333) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1010092 ERROR (Thread-4333) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1010190 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:58679
   [junit4]   2> 1010205 INFO  (jetty-launcher-1344-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010205 INFO  (jetty-launcher-1344-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010210 INFO  (jetty-launcher-1344-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010211 INFO  (jetty-launcher-1344-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010212 INFO  (jetty-launcher-1344-thread-6) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010212 INFO  (jetty-launcher-1344-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010214 INFO  (jetty-launcher-1344-thread-10) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010214 INFO  (jetty-launcher-1344-thread-9) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010215 INFO  (jetty-launcher-1344-thread-7) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010216 INFO  (jetty-launcher-1344-thread-8) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1010224 INFO  (jetty-launcher-1344-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@638e089d{/solr,null,AVAILABLE}
   [junit4]   2> 1010224 INFO  (jetty-launcher-1344-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@392d37c1{HTTP/1.1,[http/1.1]}{127.0.0.1:39950}
   [junit4]   2> 1010224 INFO  (jetty-launcher-1344-thread-1) [    ] o.e.j.s.Server Started @1013348ms
   [junit4]   2> 1010225 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39950}
   [junit4]   2> 1010225 ERROR (jetty-launcher-1344-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010225 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010225 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010225 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010225 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.302Z
   [junit4]   2> 1010228 INFO  (jetty-launcher-1344-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2ce1fa24{/solr,null,AVAILABLE}
   [junit4]   2> 1010228 INFO  (jetty-launcher-1344-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@414c0341{HTTP/1.1,[http/1.1]}{127.0.0.1:57429}
   [junit4]   2> 1010228 INFO  (jetty-launcher-1344-thread-5) [    ] o.e.j.s.Server Started @1013351ms
   [junit4]   2> 1010228 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57429}
   [junit4]   2> 1010228 ERROR (jetty-launcher-1344-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010229 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010229 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010229 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010229 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.306Z
   [junit4]   2> 1010234 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010235 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010242 INFO  (jetty-launcher-1344-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@34d975a{/solr,null,AVAILABLE}
   [junit4]   2> 1010242 INFO  (jetty-launcher-1344-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@27e74470{HTTP/1.1,[http/1.1]}{127.0.0.1:42815}
   [junit4]   2> 1010242 INFO  (jetty-launcher-1344-thread-6) [    ] o.e.j.s.Server Started @1013366ms
   [junit4]   2> 1010243 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42815}
   [junit4]   2> 1010243 ERROR (jetty-launcher-1344-thread-6) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010243 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010243 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010243 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010243 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.320Z
   [junit4]   2> 1010253 INFO  (jetty-launcher-1344-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f537b86{/solr,null,AVAILABLE}
   [junit4]   2> 1010253 INFO  (jetty-launcher-1344-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5e52702{HTTP/1.1,[http/1.1]}{127.0.0.1:53972}
   [junit4]   2> 1010253 INFO  (jetty-launcher-1344-thread-4) [    ] o.e.j.s.Server Started @1013376ms
   [junit4]   2> 1010253 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53972}
   [junit4]   2> 1010253 ERROR (jetty-launcher-1344-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010254 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010254 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010254 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010254 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.331Z
   [junit4]   2> 1010262 INFO  (jetty-launcher-1344-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2ad638e4{/solr,null,AVAILABLE}
   [junit4]   2> 1010268 INFO  (jetty-launcher-1344-thread-8) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@554a1663{/solr,null,AVAILABLE}
   [junit4]   2> 1010269 INFO  (jetty-launcher-1344-thread-8) [    ] o.e.j.s.AbstractConnector Started ServerConnector@36c66034{HTTP/1.1,[http/1.1]}{127.0.0.1:59623}
   [junit4]   2> 1010269 INFO  (jetty-launcher-1344-thread-8) [    ] o.e.j.s.Server Started @1013392ms
   [junit4]   2> 1010269 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59623}
   [junit4]   2> 1010269 ERROR (jetty-launcher-1344-thread-8) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010269 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010270 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010270 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010270 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.347Z
   [junit4]   2> 1010276 INFO  (jetty-launcher-1344-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010287 INFO  (jetty-launcher-1344-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d0ad5f9{/solr,null,AVAILABLE}
   [junit4]   2> 1010288 INFO  (jetty-launcher-1344-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d062bb3{/solr,null,AVAILABLE}
   [junit4]   2> 1010287 INFO  (jetty-launcher-1344-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@f710bc8{HTTP/1.1,[http/1.1]}{127.0.0.1:60174}
   [junit4]   2> 1010295 INFO  (jetty-launcher-1344-thread-2) [    ] o.e.j.s.Server Started @1013419ms
   [junit4]   2> 1010296 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60174}
   [junit4]   2> 1010296 ERROR (jetty-launcher-1344-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010296 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010296 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010296 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010296 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.373Z
   [junit4]   2> 1010287 INFO  (jetty-launcher-1344-thread-10) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5b704fdd{/solr,null,AVAILABLE}
   [junit4]   2> 1010297 INFO  (jetty-launcher-1344-thread-9) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d0d8feb{/solr,null,AVAILABLE}
   [junit4]   2> 1010306 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010320 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010329 INFO  (jetty-launcher-1344-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010329 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010344 INFO  (jetty-launcher-1344-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33d4016f{HTTP/1.1,[http/1.1]}{127.0.0.1:33896}
   [junit4]   2> 1010363 INFO  (jetty-launcher-1344-thread-3) [    ] o.e.j.s.Server Started @1013487ms
   [junit4]   2> 1010364 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33896}
   [junit4]   2> 1010364 ERROR (jetty-launcher-1344-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010380 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010380 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010380 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010380 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.457Z
   [junit4]   2> 1010344 INFO  (jetty-launcher-1344-thread-7) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6ca834ec{HTTP/1.1,[http/1.1]}{127.0.0.1:57502}
   [junit4]   2> 1010335 INFO  (jetty-launcher-1344-thread-10) [    ] o.e.j.s.AbstractConnector Started ServerConnector@652f01dc{HTTP/1.1,[http/1.1]}{127.0.0.1:33662}
   [junit4]   2> 1010381 INFO  (jetty-launcher-1344-thread-7) [    ] o.e.j.s.Server Started @1013504ms
   [junit4]   2> 1010381 INFO  (jetty-launcher-1344-thread-10) [    ] o.e.j.s.Server Started @1013504ms
   [junit4]   2> 1010381 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57502}
   [junit4]   2> 1010381 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33662}
   [junit4]   2> 1010381 ERROR (jetty-launcher-1344-thread-7) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010335 INFO  (jetty-launcher-1344-thread-9) [    ] o.e.j.s.AbstractConnector Started ServerConnector@242b93d0{HTTP/1.1,[http/1.1]}{127.0.0.1:56896}
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-9) [    ] o.e.j.s.Server Started @1013506ms
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56896}
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.460Z
   [junit4]   2> 1010383 ERROR (jetty-launcher-1344-thread-9) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010381 ERROR (jetty-launcher-1344-thread-10) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1010430 INFO  (jetty-launcher-1344-thread-8) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010397 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010386 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010383 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010439 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010439 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010439 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.516Z
   [junit4]   2> 1010439 INFO  (jetty-launcher-1344-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010430 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1010446 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1010446 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1010446 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-10T07:33:15.523Z
   [junit4]   2> 1010456 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010457 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010469 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1010470 INFO  (jetty-launcher-1344-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010482 INFO  (jetty-launcher-1344-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010502 INFO  (jetty-launcher-1344-thread-9) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010512 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010521 INFO  (jetty-launcher-1344-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010512 INFO  (jetty-launcher-1344-thread-5) [n:127.0.0.1:57429_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010534 INFO  (jetty-launcher-1344-thread-6) [n:127.0.0.1:42815_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010531 INFO  (jetty-launcher-1344-thread-4) [n:127.0.0.1:53972_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010545 INFO  (jetty-launcher-1344-thread-10) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58679/solr
   [junit4]   2> 1010522 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39950_solr
   [junit4]   2> 1010547 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15bf146b7200012, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1010555 INFO  (jetty-launcher-1344-thread-5) [n:127.0.0.1:57429_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57429_solr
   [junit4]   2> 1010556 INFO  (jetty-launcher-1344-thread-4) [n:127.0.0.1:53972_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53972_solr
   [junit4]   2> 1010557 INFO  (jetty-launcher-1344-thread-6) [n:127.0.0.1:42815_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42815_solr
   [junit4]   2> 1010557 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.c.Overseer Overseer (id=97937102940930057-127.0.0.1:39950_solr-n_0000000000) starting
   [junit4]   2> 1010574 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1010574 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1010610 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1010611 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1010615 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1010615 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1010615 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15bf146b720001c, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1010616 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1010620 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1010622 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1010623 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010627 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59623_solr
   [junit4]   2> 1010629 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1010630 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1010630 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1010630 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1010631 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1010632 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1010632 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1010633 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1010633 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010635 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010635 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010635 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56896_solr
   [junit4]   2> 1010636 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010637 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010637 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010637 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010638 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010638 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33896_solr
   [junit4]   2> 1010643 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60174_solr
   [junit4]   2> 1010654 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 1010654 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (7)
   [junit4]   2> 1010654 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (7)
   [junit4]   2> 1010654 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (7)
   [junit4]   2> 1010657 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 1010662 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57502_solr
   [junit4]   2> 1010663 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1010670 INFO  (zkCallback-1365-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1010670 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1010670 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 1010714 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39950_solr
   [junit4]   2> 1010715 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1010715 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1010715 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 1010742 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 1010748 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1010750 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33662_solr
   [junit4]   2> 1010751 INFO  (zkCallback-1383-thread-1-processing-n:127.0.0.1:42815_solr) [n:127.0.0.1:42815_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1010751 INFO  (zkCallback-1382-thread-1-processing-n:127.0.0.1:53972_solr) [n:127.0.0.1:53972_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1010751 INFO  (zkCallback-1363-thread-1-processing-n:127.0.0.1:57429_solr) [n:127.0.0.1:57429_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1010765 INFO  (zkCallback-1398-thread-1-processing-n:127.0.0.1:60174_solr) [n:127.0.0.1:60174_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010765 INFO  (zkCallback-1387-thread-1-processing-n:127.0.0.1:59623_solr) [n:127.0.0.1:59623_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (5)
   [junit4]   2> 1010765 INFO  (zkCallback-1392-thread-1-processing-n:127.0.0.1:56896_solr) [n:127.0.0.1:56896_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010765 INFO  (zkCallback-1396-thread-1-processing-n:127.0.0.1:33896_solr) [n:127.0.0.1:33896_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1010765 INFO  (zkCallback-1399-thread-1-processing-n:127.0.0.1:57502_solr) [n:127.0.0.1:57502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1010772 INFO  (zkCallback-1396-thread-2-processing-n:127.0.0.1:33896_solr) [n:127.0.0.1:33896_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (10)
   [junit4]   2> 1010774 INFO  (zkCallback-1398-thread-2-processing-n:127.0.0.1:60174_solr) [n:127.0.0.1:60174_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (10)
   [junit4]   2> 1010775 INFO  (zkCallback-1387-thread-2-processing-n:127.0.0.1:59623_solr) [n:127.0.0.1:59623_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (10)
   [junit4]   2> 1010794 INFO  (zkCallback-1399-thread-2-processing-n:127.0.0.1:57502_solr) [n:127.0.0.1:57502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (10)
   [junit4]   2> 1010796 INFO  (zkCallback-1392-thread-2-processing-n:127.0.0.1:56896_solr) [n:127.0.0.1:56896_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (10)
   [junit4]   2> 1010844 INFO  (zkCallback-1401-thread-1-processing-n:127.0.0.1:33662_solr) [n:127.0.0.1:33662_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 1010890 INFO  (zkCallback-1365-thread-2-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (10)
   [junit4]   2> 1011022 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60174.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011042 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60174.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011042 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60174.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011043 INFO  (jetty-launcher-1344-thread-2) [n:127.0.0.1:60174_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node2/.
   [junit4]   2> 1011100 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56896.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011107 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33662.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011113 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57502.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011175 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56896.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011175 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56896.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011176 INFO  (jetty-launcher-1344-thread-9) [n:127.0.0.1:56896_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node8/.
   [junit4]   2> 1011179 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33662.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011180 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33662.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011180 INFO  (jetty-launcher-1344-thread-5) [n:127.0.0.1:57429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57429.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011181 INFO  (jetty-launcher-1344-thread-10) [n:127.0.0.1:33662_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node7/.
   [junit4]   2> 1011182 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57502.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011183 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57502.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011184 INFO  (jetty-launcher-1344-thread-7) [n:127.0.0.1:57502_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node9/.
   [junit4]   2> 1011190 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33896.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011195 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33896.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011195 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33896.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011196 INFO  (jetty-launcher-1344-thread-3) [n:127.0.0.1:33896_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node3/.
   [junit4]   2> 1011197 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59623.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011198 INFO  (jetty-launcher-1344-thread-5) [n:127.0.0.1:57429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57429.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011198 INFO  (jetty-launcher-1344-thread-5) [n:127.0.0.1:57429_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57429.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011200 INFO  (jetty-launcher-1344-thread-5) [n:127.0.0.1:57429_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node6/.
   [junit4]   2> 1011204 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59623.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011204 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59623.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011205 INFO  (jetty-launcher-1344-thread-8) [n:127.0.0.1:59623_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node10/.
   [junit4]   2> 1011213 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011219 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011220 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39950.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011221 INFO  (jetty-launcher-1344-thread-1) [n:127.0.0.1:39950_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node1/.
   [junit4]   2> 1011223 INFO  (jetty-launcher-1344-thread-6) [n:127.0.0.1:42815_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42815.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011231 INFO  (jetty-launcher-1344-thread-6) [n:127.0.0.1:42815_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42815.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011231 INFO  (jetty-launcher-1344-thread-6) [n:127.0.0.1:42815_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42815.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011239 INFO  (jetty-launcher-1344-thread-6) [n:127.0.0.1:42815_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node5/.
   [junit4]   2> 1011251 INFO  (jetty-launcher-1344-thread-4) [n:127.0.0.1:53972_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53972.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011280 INFO  (jetty-launcher-1344-thread-4) [n:127.0.0.1:53972_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53972.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011280 INFO  (jetty-launcher-1344-thread-4) [n:127.0.0.1:53972_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53972.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4194bd1
   [junit4]   2> 1011281 INFO  (jetty-launcher-1344-thread-4) [n:127.0.0.1:53972_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_D075F912027B4DA5-001/tempDir-001/node4/.
   [junit4]   2> 1012015 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 1012016 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58679/solr ready
   [junit4]   2> 1012457 INFO  (qtp2061209599-11822) [n:127.0.0.1:53972_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&name=test_col&action=CREATE&numShards=5&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1012459 INFO  (OverseerThreadFactory-3653-thread-1-processing-n:127.0.0.1:39950_solr) [n:127.0.0.1:39950_solr    ] o.a.s.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 1012581 INFO  (qtp1687932765-11857) [n:127.0.0.1:56896_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard2_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1012585 INFO  (qtp1687932765-11857) [n:127.0.0.1:56896_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1012591 INFO  (qtp2061209599-11820) [n:127.0.0.1:53972_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard5_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard5&wt=javabin&version=2
   [junit4]   2> 1012591 INFO  (qtp1845385632-11777) [n:127.0.0.1:60174_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard4_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1012591 INFO  (qtp1845385632-11777) [n:127.0.0.1:60174_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1012591 INFO  (qtp2061209599-11820) [n:127.0.0.1:53972_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1012597 INFO  (qtp1350071380-11814) [n:127.0.0.1:42815_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard3_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1012597 INFO  (qtp1350071380-11814) [n:127.0.0.1:42815_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1012623 INFO  (qtp630700487-11852) [n:127.0.0.1:33662_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard1_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1012623 INFO  (qtp784325897-11803) [n:127.0.0.1:57429_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard1_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1012624 INFO  (qtp784325897-11803) [n:127.0.0.1:57429_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1012624 INFO  (qtp792432739-11881) [n:127.0.0.1:57502_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog

[...truncated too long message...]

g.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest.teardownClass(HdfsTlogReplayBufferedWhileIndexingTest.java:52)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> 3706318 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D075F912027B4DA5]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D075F912027B4DA5-001
   [junit4]   2> May 10, 2017 8:18:11 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=no-NO, timezone=Pacific/Midway
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=295598512,total=521142272
   [junit4]   2> NOTE: All tests run in this JVM: [UnloadDistributedZkTest, ShowFileRequestHandlerTest, TestSolrCloudWithDelegationTokens, SaslZkACLProviderTest, TestFilteredDocIdSet, SynonymTokenizerTest, ConcurrentDeleteAndCreateCollectionTest, BinaryUpdateRequestHandlerTest, SolrPluginUtilsTest, TestBinaryField, OpenExchangeRatesOrgProviderTest, SimpleFacetsTest, CircularListTest, TestFieldTypeCollectionResource, BlockCacheTest, CloudExitableDirectoryReaderTest, DateFieldTest, TestStandardQParsers, TestStressReorder, OutputWriterTest, AtomicUpdateProcessorFactoryTest, SolrCoreCheckLockOnStartupTest, MultiThreadedOCPTest, TestTolerantSearch, HdfsThreadLeakTest, TestSlowCompositeReaderWrapper, TestDelegationWithHadoopAuth, TestNonDefinedSimilarityFactory, CdcrVersionReplicationTest, TestSegmentSorting, TestUniqueKeyFieldResource, FieldMutatingUpdateProcessorTest, DistributedSuggestComponentTest, SolrCloudExampleTest, SolrIndexConfigTest, TestMacros, CreateCollectionCleanupTest, TestConfigOverlay, BasicDistributedZk2Test, ChaosMonkeyNothingIsSafeTest, HighlighterConfigTest, TestClusterProperties, TestLuceneMatchVersion, TestSolrDeletionPolicy1, TestCollationField, SuggesterTest, HttpSolrCallGetCoreTest, CdcrReplicationDistributedZkTest, RuleEngineTest, HardAutoCommitTest, BasicAuthStandaloneTest, DeleteLastCustomShardedReplicaTest, TestExtendedDismaxParser, TestSchemaResource, PluginInfoTest, SuggesterWFSTTest, HttpPartitionTest, TestFieldCache, PreAnalyzedUpdateProcessorTest, TestFieldCacheSortRandom, TestTolerantUpdateProcessorCloud, TestSolrQueryResponse, TestMiniSolrCloudClusterSSL, ActionThrottleTest, V2ApiIntegrationTest, MergeStrategyTest, TestDocSet, SimpleMLTQParserTest, TestQuerySenderNoQuery, HdfsRecoveryZkTest, TestRandomRequestDistribution, PropertiesRequestHandlerTest, LeaderFailureAfterFreshStartTest, DistribDocExpirationUpdateProcessorTest, ClassificationUpdateProcessorIntegrationTest, TestClassNameShortening, TestComplexPhraseQParserPlugin, StatsComponentTest, TestConfig, TestShardHandlerFactory, TestDeleteCollectionOnDownNodes, TestEmbeddedSolrServerSchemaAPI, HdfsWriteToMultipleCollectionsTest, TestPKIAuthenticationPlugin, UpdateParamsTest, DirectUpdateHandlerOptimizeTest, TestWriterPerf, OverseerTaskQueueTest, TestGeoJSONResponseWriter, TestDistributedMissingSort, RecoveryZkTest, JavabinLoaderTest, SpellCheckCollatorWithCollapseTest, MetricUtilsTest, DebugComponentTest, HdfsDirectoryFactoryTest, TestConfigSetsAPIExclusivity, AutoCommitTest, DateRangeFieldTest, TestExclusionRuleCollectionAccess, TestCSVResponseWriter, DistributedFacetPivotLongTailTest, TestSortByMinMaxFunction, TestRTimerTree, TestStressCloudBlindAtomicUpdates, TestConfigSets, TestBulkSchemaConcurrent, TestHdfsUpdateLog, CdcrUpdateLogTest, RemoteQueryErrorTest, TestInitParams, TestMissingGroups, TestSort, TestSearcherReuse, SharedFSAutoReplicaFailoverUtilsTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, CurrencyFieldOpenExchangeTest, CurrencyFieldXmlFileTest, AnalysisAfterCoreReloadTest, CoreAdminHandlerTest, SuggesterTSTTest, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, SolrInfoBeanTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, TestValueSourceCache, TestIndexingPerformance, FastVectorHighlighterTest, LoggingHandlerTest, TestJmxIntegration, UniqFieldsUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, PingRequestHandlerTest, TestLFUCache, CleanupOldIndexTest, CollectionReloadTest, CollectionTooManyReplicasTest, DeleteInactiveReplicaTest, DeleteReplicaTest, DeleteStatusTest, DistributedVersionInfoTest, LeaderInitiatedRecoveryOnCommitTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, OverseerModifyCollectionTest, PeerSyncReplicationTest, RecoveryAfterSoftCommitTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudPivotFacet, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry, TestLockTree, TestMiniSolrCloudCluster, TestReplicaProperties, TestSolrCloudWithSecureImpersonation, TestStressLiveNodes, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZkTest, HdfsTlogReplayBufferedWhileIndexingTest]
   [junit4] Completed [651/710 (3!)] on J2 in 352.68s, 1 test, 1 failure <<< FAILURES!

[...truncated 43573 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1297 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1297/

3 tests failed.
FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
Unable to restart (#3): CloudJettyRunner [url=http://127.0.0.1:41853/collection1]

Stack Trace:
java.lang.AssertionError: Unable to restart (#3): CloudJettyRunner [url=http://127.0.0.1:41853/collection1]
        at __randomizedtesting.SeedInfo.seed([2C408293782D03E8:A414BD49D6D16E10]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:103)
        at org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([2C408293782D03E8:DBEF4BB885A4E9C3]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:40813/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:52500/solr/test_col_shard3_replica1: Server Error    request: http://127.0.0.1:52500/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A40813%2Fsolr%2Ftest_col_shard2_replica1%2F&wt=javabin&version=2 Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:35019/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@5dcd3caf

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:40813/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:52500/solr/test_col_shard3_replica1: Server Error



request: http://127.0.0.1:52500/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A40813%2Fsolr%2Ftest_col_shard2_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:35019/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@5dcd3caf
        at __randomizedtesting.SeedInfo.seed([2C408293782D03E8:1A54E0D5F27039F9]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:40813/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:52500/solr/test_col_shard3_replica1: Server Error



request: http://127.0.0.1:52500/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A40813%2Fsolr%2Ftest_col_shard2_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:35019/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@5dcd3caf
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40813/solr/test_col: Async exception during distributed update: Error from server at http://127.0.0.1:52500/solr/test_col_shard3_replica1: Server Error



request: http://127.0.0.1:52500/solr/test_col_shard3_replica1/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A40813%2Fsolr%2Ftest_col_shard2_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: http://127.0.0.1:35019/solr/test_col_shard3_replica2/ update: org.apache.solr.client.solrj.request.UpdateRequest@5dcd3caf
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:641)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:281)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:270)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
        ... 6 more




Build Log:
[...truncated 11959 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/init-core-data-001
   [junit4]   2> 587247 WARN  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 587247 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 587249 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=SSL overhead seems to cause OutOfMemory when stress testing)
   [junit4]   2> 587249 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 10 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001
   [junit4]   2> 587250 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 587250 INFO  (Thread-5446) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 587250 INFO  (Thread-5446) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 587253 ERROR (Thread-5446) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 587350 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:57258
   [junit4]   2> 587358 INFO  (jetty-launcher-762-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587358 INFO  (jetty-launcher-762-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587361 INFO  (jetty-launcher-762-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587362 INFO  (jetty-launcher-762-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587367 INFO  (jetty-launcher-762-thread-6) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587368 INFO  (jetty-launcher-762-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@73360440{/solr,null,AVAILABLE}
   [junit4]   2> 587369 INFO  (jetty-launcher-762-thread-5) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587369 INFO  (jetty-launcher-762-thread-7) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587370 INFO  (jetty-launcher-762-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5ef45876{/solr,null,AVAILABLE}
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@61449aed{HTTP/1.1,[http/1.1]}{127.0.0.1:52500}
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.e.j.s.Server Started @590243ms
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1570cde4{HTTP/1.1,[http/1.1]}{127.0.0.1:35019}
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52500}
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.e.j.s.Server Started @590243ms
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35019}
   [junit4]   2> 587371 ERROR (jetty-launcher-762-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587371 ERROR (jetty-launcher-762-thread-6) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.200Z
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.200Z
   [junit4]   2> 587371 INFO  (jetty-launcher-762-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e279aa6{/solr,null,AVAILABLE}
   [junit4]   2> 587373 INFO  (jetty-launcher-762-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3b20a1df{/solr,null,AVAILABLE}
   [junit4]   2> 587373 INFO  (jetty-launcher-762-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2dbb763{HTTP/1.1,[http/1.1]}{127.0.0.1:47856}
   [junit4]   2> 587373 INFO  (jetty-launcher-762-thread-1) [    ] o.e.j.s.Server Started @590246ms
   [junit4]   2> 587383 INFO  (jetty-launcher-762-thread-8) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587383 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=47856}
   [junit4]   2> 587384 ERROR (jetty-launcher-762-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587375 INFO  (jetty-launcher-762-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2f6498dd{HTTP/1.1,[http/1.1]}{127.0.0.1:35260}
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-4) [    ] o.e.j.s.Server Started @590257ms
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35260}
   [junit4]   2> 587383 INFO  (jetty-launcher-762-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5da71b03{/solr,null,AVAILABLE}
   [junit4]   2> 587384 ERROR (jetty-launcher-762-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587375 INFO  (jetty-launcher-762-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4579d9fd{/solr,null,AVAILABLE}
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.213Z
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587384 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.213Z
   [junit4]   2> 587385 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587386 INFO  (jetty-launcher-762-thread-9) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587391 INFO  (jetty-launcher-762-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7a204eaa{HTTP/1.1,[http/1.1]}{127.0.0.1:40386}
   [junit4]   2> 587391 INFO  (jetty-launcher-762-thread-2) [    ] o.e.j.s.Server Started @590264ms
   [junit4]   2> 587391 INFO  (jetty-launcher-762-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1a3d6535{HTTP/1.1,[http/1.1]}{127.0.0.1:35465}
   [junit4]   2> 587391 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40386}
   [junit4]   2> 587392 INFO  (jetty-launcher-762-thread-5) [    ] o.e.j.s.Server Started @590264ms
   [junit4]   2> 587392 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35465}
   [junit4]   2> 587392 ERROR (jetty-launcher-762-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587392 ERROR (jetty-launcher-762-thread-5) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587392 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587392 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587392 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587392 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.221Z
   [junit4]   2> 587394 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587394 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587394 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587394 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.223Z
   [junit4]   2> 587395 INFO  (jetty-launcher-762-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@10dc7cdf{/solr,null,AVAILABLE}
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.e.j.s.AbstractConnector Started ServerConnector@106b1b9c{HTTP/1.1,[http/1.1]}{127.0.0.1:40813}
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.e.j.s.Server Started @590269ms
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40813}
   [junit4]   2> 587396 ERROR (jetty-launcher-762-thread-7) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587396 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.225Z
   [junit4]   2> 587399 INFO  (jetty-launcher-762-thread-10) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 587402 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587409 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587409 INFO  (jetty-launcher-762-thread-9) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@120456ba{/solr,null,AVAILABLE}
   [junit4]   2> 587409 INFO  (jetty-launcher-762-thread-9) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7549d60a{HTTP/1.1,[http/1.1]}{127.0.0.1:55655}
   [junit4]   2> 587409 INFO  (jetty-launcher-762-thread-9) [    ] o.e.j.s.Server Started @590282ms
   [junit4]   2> 587409 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55655}
   [junit4]   2> 587410 ERROR (jetty-launcher-762-thread-9) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587410 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587410 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587410 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587410 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.239Z
   [junit4]   2> 587416 INFO  (jetty-launcher-762-thread-8) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@44b4d9e1{/solr,null,AVAILABLE}
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7ebfd720{HTTP/1.1,[http/1.1]}{127.0.0.1:47860}
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.e.j.s.Server Started @590289ms
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=47860}
   [junit4]   2> 587417 ERROR (jetty-launcher-762-thread-8) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587417 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.246Z
   [junit4]   2> 587420 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587426 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587427 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587433 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587433 INFO  (jetty-launcher-762-thread-10) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@cf30135{/solr,null,AVAILABLE}
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.e.j.s.AbstractConnector Started ServerConnector@f046e63{HTTP/1.1,[http/1.1]}{127.0.0.1:33390}
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.e.j.s.Server Started @590307ms
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33390}
   [junit4]   2> 587434 ERROR (jetty-launcher-762-thread-10) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 587434 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-11T08:38:15.263Z
   [junit4]   2> 587438 INFO  (jetty-launcher-762-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587443 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15bf6a895280005, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 587449 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587461 INFO  (jetty-launcher-762-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587463 INFO  (jetty-launcher-762-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587463 INFO  (jetty-launcher-762-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587465 INFO  (jetty-launcher-762-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587468 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587487 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 587493 INFO  (jetty-launcher-762-thread-9) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587494 INFO  (jetty-launcher-762-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587497 INFO  (jetty-launcher-762-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587501 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15bf6a89528000f, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 587503 INFO  (jetty-launcher-762-thread-10) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587507 INFO  (jetty-launcher-762-thread-8) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57258/solr
   [junit4]   2> 587534 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587534 INFO  (jetty-launcher-762-thread-1) [n:127.0.0.1:47856_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587535 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35019_solr
   [junit4]   2> 587538 INFO  (jetty-launcher-762-thread-1) [n:127.0.0.1:47856_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47856_solr
   [junit4]   2> 587539 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.c.Overseer Overseer (id=97943020835962900-127.0.0.1:35019_solr-n_0000000000) starting
   [junit4]   2> 587543 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 587547 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 587550 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 587552 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 587552 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 587553 INFO  (jetty-launcher-762-thread-2) [n:127.0.0.1:40386_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587555 INFO  (jetty-launcher-762-thread-9) [n:127.0.0.1:55655_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587563 INFO  (jetty-launcher-762-thread-2) [n:127.0.0.1:40386_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40386_solr
   [junit4]   2> 587565 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 587566 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587568 INFO  (jetty-launcher-762-thread-9) [n:127.0.0.1:55655_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55655_solr
   [junit4]   2> 587569 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 587570 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 587570 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 587570 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 587574 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 587575 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 587579 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 587580 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 587594 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587594 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587607 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587606 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587606 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587610 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40813_solr
   [junit4]   2> 587610 INFO  (zkCallback-807-thread-2-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 587615 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587615 INFO  (zkCallback-814-thread-1-processing-n:127.0.0.1:35260_solr) [n:127.0.0.1:35260_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587616 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 587616 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 587616 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 587616 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 587616 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 587616 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 587627 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 587628 INFO  (zkCallback-816-thread-1-processing-n:127.0.0.1:47860_solr) [n:127.0.0.1:47860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 587629 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587631 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587632 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35465_solr
   [junit4]   2> 587636 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35260_solr
   [junit4]   2> 587637 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 587637 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 587637 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 587637 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 587637 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 587639 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 587640 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587642 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 587646 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 587646 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 587646 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 587646 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 587654 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 587654 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47860_solr
   [junit4]   2> 587655 INFO  (zkCallback-816-thread-2-processing-n:127.0.0.1:47860_solr) [n:127.0.0.1:47860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (6)
   [junit4]   2> 587658 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35019_solr
   [junit4]   2> 587659 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 587659 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33390_solr
   [junit4]   2> 587662 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 587662 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 587662 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 587662 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 587662 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 587662 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 587666 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 587666 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 587666 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 587666 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 587666 INFO  (zkCallback-814-thread-2-processing-n:127.0.0.1:35260_solr) [n:127.0.0.1:35260_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (8)
   [junit4]   2> 587668 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 587668 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 587668 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52500_solr
   [junit4]   2> 587668 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 587668 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 587668 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 587669 INFO  (zkCallback-816-thread-2-processing-n:127.0.0.1:47860_solr) [n:127.0.0.1:47860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (9)
   [junit4]   2> 587671 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 587671 INFO  (zkCallback-816-thread-2-processing-n:127.0.0.1:47860_solr) [n:127.0.0.1:47860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 587671 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (10)
   [junit4]   2> 587686 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 587686 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 587686 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 587695 INFO  (zkCallback-818-thread-1-processing-n:127.0.0.1:33390_solr) [n:127.0.0.1:33390_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (10)
   [junit4]   2> 587695 INFO  (zkCallback-811-thread-1-processing-n:127.0.0.1:52500_solr) [n:127.0.0.1:52500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (6)
   [junit4]   2> 587698 INFO  (jetty-launcher-762-thread-1) [n:127.0.0.1:47856_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47856.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587702 INFO  (jetty-launcher-762-thread-1) [n:127.0.0.1:47856_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47856.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587703 INFO  (jetty-launcher-762-thread-1) [n:127.0.0.1:47856_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47856.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587704 INFO  (jetty-launcher-762-thread-1) [n:127.0.0.1:47856_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node1/.
   [junit4]   2> 587738 INFO  (jetty-launcher-762-thread-2) [n:127.0.0.1:40386_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40386.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587743 INFO  (jetty-launcher-762-thread-2) [n:127.0.0.1:40386_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40386.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587743 INFO  (jetty-launcher-762-thread-2) [n:127.0.0.1:40386_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40386.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587744 INFO  (jetty-launcher-762-thread-2) [n:127.0.0.1:40386_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node2/.
   [junit4]   2> 587758 INFO  (zkCallback-811-thread-2-processing-n:127.0.0.1:52500_solr) [n:127.0.0.1:52500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (10)
   [junit4]   2> 587758 INFO  (zkCallback-814-thread-2-processing-n:127.0.0.1:35260_solr) [n:127.0.0.1:35260_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (10)
   [junit4]   2> 587760 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 587830 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47860.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587857 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35465.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587861 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47860.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587861 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47860.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587867 INFO  (jetty-launcher-762-thread-8) [n:127.0.0.1:47860_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node8/.
   [junit4]   2> 587885 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35260.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587894 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35465.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587894 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35465.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587898 INFO  (jetty-launcher-762-thread-5) [n:127.0.0.1:35465_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node5/.
   [junit4]   2> 587916 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35260.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587916 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35260.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587917 INFO  (jetty-launcher-762-thread-9) [n:127.0.0.1:55655_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55655.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587917 INFO  (jetty-launcher-762-thread-4) [n:127.0.0.1:35260_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node4/.
   [junit4]   2> 587928 INFO  (jetty-launcher-762-thread-9) [n:127.0.0.1:55655_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55655.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587928 INFO  (jetty-launcher-762-thread-9) [n:127.0.0.1:55655_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55655.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587929 INFO  (jetty-launcher-762-thread-9) [n:127.0.0.1:55655_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node9/.
   [junit4]   2> 587931 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40813.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587931 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35019.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587935 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52500.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587936 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40813.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587936 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40813.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587937 INFO  (jetty-launcher-762-thread-7) [n:127.0.0.1:40813_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node7/.
   [junit4]   2> 587937 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35019.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587940 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52500.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587941 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52500.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587941 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33390.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587941 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35019.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587942 INFO  (jetty-launcher-762-thread-3) [n:127.0.0.1:52500_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node3/.
   [junit4]   2> 587942 INFO  (jetty-launcher-762-thread-6) [n:127.0.0.1:35019_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node6/.
   [junit4]   2> 587946 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33390.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587947 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33390.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 587948 INFO  (jetty-launcher-762-thread-10) [n:127.0.0.1:33390_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node10/.
   [junit4]   2> 588576 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 588577 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[2C408293782D03E8]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:57258/solr ready
   [junit4]   2> 589042 INFO  (qtp1969498763-8951) [n:127.0.0.1:47856_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&name=test_col&action=CREATE&numShards=5&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 589045 INFO  (OverseerThreadFactory-2008-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 589166 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard1_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard1&wt=javabin&version=2
   [junit4]   2> 589166 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard1_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard1&wt=javabin&version=2
   [junit4]   2> 589167 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589167 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589174 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard4_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard4&wt=javabin&version=2
   [junit4]   2> 589174 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589175 INFO  (qtp1613625346-8969) [n:127.0.0.1:35019_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard3_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard3&wt=javabin&version=2
   [junit4]   2> 589175 INFO  (qtp1613625346-8969) [n:127.0.0.1:35019_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589176 INFO  (qtp1745181850-9002) [n:127.0.0.1:40813_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard2_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard2&wt=javabin&version=2
   [junit4]   2> 589176 INFO  (qtp1745181850-9002) [n:127.0.0.1:40813_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589177 INFO  (qtp1095611779-9030) [n:127.0.0.1:47860_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard5_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard5&wt=javabin&version=2
   [junit4]   2> 589177 INFO  (qtp1095611779-9030) [n:127.0.0.1:47860_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589178 INFO  (qtp109602435-9034) [n:127.0.0.1:55655_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard5_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard5&wt=javabin&version=2
   [junit4]   2> 589178 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard3_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard3&wt=javabin&version=2
   [junit4]   2> 589180 INFO  (qtp109602435-9034) [n:127.0.0.1:55655_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589180 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589182 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard2_replica2&action=CREATE&numShards=5&collection=test_col&shard=shard2&wt=javabin&version=2
   [junit4]   2> 589182 INFO  (qtp938004591-8993) [n:127.0.0.1:35465_solr    ] o.a.s.h.a.CoreAdminOperation core create command property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&qt=/admin/cores&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&newCollection=true&name=test_col_shard4_replica1&action=CREATE&numShards=5&collection=test_col&shard=shard4&wt=javabin&version=2
   [junit4]   2> 589182 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589182 INFO  (qtp938004591-8993) [n:127.0.0.1:35465_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 589300 INFO  (zkCallback-814-thread-2-processing-n:127.0.0.1:35260_solr) [n:127.0.0.1:35260_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-800-thread-1-processing-n:127.0.0.1:35019_solr) [n:127.0.0.1:35019_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-791-thread-1-processing-n:127.0.0.1:47856_solr) [n:127.0.0.1:47856_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-816-thread-2-processing-n:127.0.0.1:47860_solr) [n:127.0.0.1:47860_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-808-thread-1-processing-n:127.0.0.1:40813_solr) [n:127.0.0.1:40813_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-811-thread-2-processing-n:127.0.0.1:52500_solr) [n:127.0.0.1:52500_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-802-thread-1-processing-n:127.0.0.1:55655_solr) [n:127.0.0.1:55655_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-818-thread-1-processing-n:127.0.0.1:33390_solr) [n:127.0.0.1:33390_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-807-thread-1-processing-n:127.0.0.1:35465_solr) [n:127.0.0.1:35465_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 589300 INFO  (zkCallback-803-thread-1-processing-n:127.0.0.1:40386_solr) [n:127.0.0.1:40386_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [10])
   [junit4]   2> 590191 WARN  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590192 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590195 WARN  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590195 WARN  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590196 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590196 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590197 WARN  (qtp109602435-9034) [n:127.0.0.1:55655_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590198 INFO  (qtp109602435-9034) [n:127.0.0.1:55655_solr c:test_col s:shard5  x:test_col_shard5_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590201 WARN  (qtp1613625346-8969) [n:127.0.0.1:35019_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590202 INFO  (qtp1613625346-8969) [n:127.0.0.1:35019_solr c:test_col s:shard3  x:test_col_shard3_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590211 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.s.IndexSchema [test_col_shard4_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 590229 WARN  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590230 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590237 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.s.IndexSchema [test_col_shard1_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 590239 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 590239 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 590243 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.s.IndexSchema [test_col_shard1_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 590243 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.s.IndexSchema [test_col_shard3_replica1] Schema name=minimal-atomic-stress
   [junit4]   2> 590245 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 590245 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 590246 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 590246 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard3_replica1' using configuration from collection test_col, trusted=true
   [junit4]   2> 590246 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_47856.solr.core.test_col.shard1.replica2' (registry 'solr.core.test_col.shard1.replica2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 590246 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52500.solr.core.test_col.shard3.replica1' (registry 'solr.core.test_col.shard3.replica1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 590246 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 590246 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 590246 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.c.SolrCore [[test_col_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node1/test_col_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node1/./test_col_shard1_replica2/data/]
   [junit4]   2> 590246 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.c.SolrCore [[test_col_shard3_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node3/test_col_shard3_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node3/./test_col_shard3_replica1/data/]
   [junit4]   2> 590248 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35260.solr.core.test_col.shard1.replica1' (registry 'solr.core.test_col.shard1.replica1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 590248 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 590248 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.c.SolrCore [[test_col_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node4/test_col_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node4/./test_col_shard1_replica1/data/]
   [junit4]   2> 590250 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 590250 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard4_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 590250 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40386.solr.core.test_col.shard4.replica2' (registry 'solr.core.test_col.shard4.replica2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@71b80374
   [junit4]   2> 590250 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 590251 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.SolrCore [[test_col_shard4_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node2/test_col_shard4_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_2C408293782D03E8-001/tempDir-001/node2/./test_col_shard4_replica2/data/]
   [junit4]   2> 590253 INFO  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@726fc8ec
   [junit4]   2> 590255 INFO  (qtp2005573935-8950) [n:127.0.0.1:52500_solr c:test_col s:shard3  x:test_col_shard3_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6870162e
   [junit4]   2> 590255 WARN  (qtp1118240897-9051) [n:127.0.0.1:33390_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590256 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590257 WARN  (qtp1095611779-9030) [n:127.0.0.1:47860_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590259 WARN  (qtp1500356904-8963) [n:127.0.0.1:40386_solr c:test_col s:shard4  x:test_col_shard4_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 590261 INFO  (qtp351094396-8985) [n:127.0.0.1:35260_solr c:test_col s:shard1  x:test_col_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@27d5eb8c
   [junit4]   2> 590265 WARN  (qtp938004591-8993) [n:127.0.0.1:35465_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590266 INFO  (qtp938004591-8993) [n:127.0.0.1:35465_solr c:test_col s:shard4  x:test_col_shard4_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590272 INFO  (qtp1095611779-9030) [n:127.0.0.1:47860_solr c:test_col s:shard5  x:test_col_shard5_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590272 INFO  (qtp1969498763-8942) [n:127.0.0.1:47856_solr c:test_col s:shard1  x:test_col_shard1_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7b6d549
   [junit4]   2> 590276 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.s.IndexSchema [test_col_shard2_replica2] Schema name=minimal-atomic-stress
   [junit4]   2> 590268 WARN  (qtp1745181850-9002) [n:127.0.0.1:40813_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 590278 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.s.IndexSchema Loaded schema minimal-atomic-stress/1.6 with uniqueid field id
   [junit4]   2> 590278 INFO  (qtp1745181850-9002) [n:127.0.0.1:40813_solr c:test_col s:shard2  x:test_col_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 590279 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_solr c:test_col s:shard2  x:test_col_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica2' using configuration from collection test_col, trusted=true
   [junit4]   2> 590279 INFO  (qtp1118240897-9051) [n:127.0.0.1:33390_

[...truncated too long message...]

SourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.core.HdfsDirectoryFactoryTest.teardownClass(HdfsDirectoryFactoryTest.java:64)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> 3613092 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[2C408293782D03E8]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_2C408293782D03E8-001
   [junit4]   2> May 11, 2017 9:28:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=534, maxMBSortInHeap=7.552364998350498, sim=RandomSimilarity(queryNorm=false): {}, locale=tr-TR, timezone=America/Glace_Bay
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=334569680,total=522715136
   [junit4]   2> NOTE: All tests run in this JVM: [SecurityConfHandlerTest, QueryEqualityTest, UniqFieldsUpdateProcessorFactoryTest, TestShortCircuitedRequests, TestCollectionAPI, TestReplicationHandlerBackup, ResponseHeaderTest, HdfsNNFailoverTest, TestSQLHandler, ClassificationUpdateProcessorTest, RequiredFieldsTest, FastVectorHighlighterTest, TestRangeQuery, TestLMJelinekMercerSimilarityFactory, CollectionReloadTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, AnalysisAfterCoreReloadTest, BaseCdcrDistributedZkTest, TestRestManager, TestPathTrie, SignatureUpdateProcessorFactoryTest, JSONWriterTest, PeerSyncTest, TestSolrCloudSnapshots, OverseerRolesTest, TestRebalanceLeaders, TestHdfsCloudBackupRestore, TestIntervalFaceting, CdcrReplicationHandlerTest, TestHighlightDedupGrouping, TestSubQueryTransformerDistrib, RAMDirectoryFactoryTest, RulesTest, TestCrossCoreJoin, BasicDistributedZkTest, TestDFRSimilarityFactory, HighlighterMaxOffsetTest, OverriddenZkACLAndCredentialsProvidersTest, RegexBoostProcessorTest, NumericFieldsTest, TestMiniSolrCloudCluster, TestRandomCollapseQParserPlugin, DistributedFacetExistsSmallTest, LeaderElectionIntegrationTest, TestJettySolrRunner, TestGraphTermsQParserPlugin, TestAnalyzeInfixSuggestions, XsltUpdateRequestHandlerTest, TestLegacyNumericRangeQueryBuilder, TestRequestStatusCollectionAPI, TestValueSourceCache, ResponseLogComponentTest, CustomCollectionTest, TolerantUpdateProcessorTest, PolyFieldTest, DocValuesNotIndexedTest, TestRemoteStreaming, BlockDirectoryTest, TestRandomDVFaceting, QueryElevationComponentTest, CacheHeaderTest, DirectoryFactoryTest, LeaderElectionContextKeyTest, TestNumericTerms32, JsonValidatorTest, ZkNodePropsTest, UninvertDocValuesMergePolicyTest, DefaultValueUpdateProcessorTest, TestPerFieldSimilarityWithDefaultOverride, TestManagedStopFilterFactory, TestFunctionQuery, AliasIntegrationTest, HdfsTlogReplayBufferedWhileIndexingTest, SearchHandlerTest, CurrencyFieldOpenExchangeTest, CursorPagingTest, TestCopyFieldCollectionResource, TestLazyCores, VersionInfoTest, TestDistributedSearch, HLLUtilTest, TestStressInPlaceUpdates, TestJoin, AtomicUpdatesTest, TestInPlaceUpdatesDistrib, FileBasedSpellCheckerTest, TestConfigSetImmutable, LeaderInitiatedRecoveryOnCommitTest, DeleteStatusTest, ShardSplitTest, SolrShardReporterTest, MissingSegmentRecoveryTest, TestCollapseQParserPlugin, TestJmxIntegration, SolrJmxReporterTest, DateMathParserTest, PKIAuthenticationIntegrationTest, TestDocumentBuilder, NotRequiredUniqueKeyTest, TestSimpleQParserPlugin, TestFieldTypeResource, PeerSyncReplicationTest, TestPivotHelperCode, TestSuggestSpellingConverter, TestFieldCollectionResource, TestStressUserVersions, TestPseudoReturnFields, TestLeaderInitiatedRecoveryThread, MultiTermTest, TestFoldingMultitermQuery, BJQParserTest, PathHierarchyTokenizerFactoryTest, TestRealTimeGet, TestReload, StatsComponentTest, BasicFunctionalityTest, TestBadConfig, SoftAutoCommitTest, TestUpdate, TestAtomicUpdateErrorCases, SolrInfoBeanTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, StandardRequestHandlerTest, TestWriterPerf, TestQueryTypes, TestOmitPositions, RequestHandlersTest, TestSolrQueryParser, TestCollationField, ReturnFieldsTest, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, AlternateDirectoryTest, TestSolrIndexConfig, CopyFieldTest, TestSolrDeletionPolicy2, SampleTest, TestDocSet, TestBinaryField, TestSearchPerf, MinimalSchemaTest, TestConfig, TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnectionTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PreAnalyzedFieldTest, DateFieldTest, TestSolrJ, TestUtils, UUIDFieldTest, CircularListTest, TestRTGBase, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, AsyncCallRequestStatusResponseTest, CollectionStateFormat2Test, CollectionsAPIAsyncDistributedZkTest, ConcurrentDeleteAndCreateCollectionTest, DistribJoinFromCollectionTest, LeaderFailoverAfterPartitionTest, NodeMutatorTest, RecoveryAfterSoftCommitTest, SSLMigrationTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestLeaderElectionZkExpiry, TestReplicaProperties, TestSegmentSorting, TestStressLiveNodes, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZkTest, ConfigureRecoveryStrategyTest, HdfsDirectoryFactoryTest]
   [junit4] Completed [544/710 (3!)] on J0 in 13.36s, 3 tests, 1 error <<< FAILURES!

[...truncated 43887 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1298 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1298/

1 tests failed.
FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([B9436D789A6A2D90:4EECA45367E3C7BB]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more




Build Log:
[...truncated 13301 lines...]
   [junit4] Suite: org.apache.solr.core.HdfsDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/init-core-data-001
   [junit4]   2> 3492116 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 3492116 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 3492119 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3492180 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3492195 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3492200 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3492214 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_51384_hdfs____k7g4cd/webapp
   [junit4]   2> 3492665 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51384
   [junit4]   2> 3492736 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3492738 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3492760 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_60218_datanode____.afzr22/webapp
   [junit4]   2> 3493167 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60218
   [junit4]   2> 3493213 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3493214 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3493229 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36133_datanode____.3ktq17/webapp
   [junit4]   2> 3493305 INFO  (IPC Server handler 3 on 53614) [    ] BlockStateChange BLOCK* processReport: from storage DS-5417c3bd-f1e7-4cb0-a08e-20be207e7bee node DatanodeRegistration(127.0.0.1:49771, datanodeUuid=38f8f73f-72a3-4191-ba24-b21e75984763, infoPort=48208, infoSecurePort=0, ipcPort=48992, storageInfo=lv=-56;cid=testClusterID;nsid=1779180660;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 3493305 INFO  (IPC Server handler 3 on 53614) [    ] BlockStateChange BLOCK* processReport: from storage DS-f388a628-6c5e-42f1-a941-c613ebf40910 node DatanodeRegistration(127.0.0.1:49771, datanodeUuid=38f8f73f-72a3-4191-ba24-b21e75984763, infoPort=48208, infoSecurePort=0, ipcPort=48992, storageInfo=lv=-56;cid=testClusterID;nsid=1779180660;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3493676 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36133
   [junit4]   2> 3493796 INFO  (IPC Server handler 3 on 53614) [    ] BlockStateChange BLOCK* processReport: from storage DS-8e976fbd-169e-45e7-8c0a-66c0a1b7d8ec node DatanodeRegistration(127.0.0.1:60123, datanodeUuid=d2168978-e0c3-471d-8dce-6dd54416f792, infoPort=45678, infoSecurePort=0, ipcPort=58909, storageInfo=lv=-56;cid=testClusterID;nsid=1779180660;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3493796 INFO  (IPC Server handler 3 on 53614) [    ] BlockStateChange BLOCK* processReport: from storage DS-00975e1f-8675-4514-87c8-5a37214e0973 node DatanodeRegistration(127.0.0.1:60123, datanodeUuid=d2168978-e0c3-471d-8dce-6dd54416f792, infoPort=45678, infoSecurePort=0, ipcPort=58909, storageInfo=lv=-56;cid=testClusterID;nsid=1779180660;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3493947 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[B9436D789A6A2D90]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCleanupOldIndexDirectories
   [junit4]   2> 3493947 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53614/solr1
   [junit4]   2> 3493947 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3493964 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Found 1 old index directories to clean-up under hdfs://localhost:53614/solr1/mock/data afterReload=false
   [junit4]   2> 3493965 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Deleted old index directory hdfs://localhost:53614/solr1/mock/data/index.20170513042927015
   [junit4]   2> 3493966 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[B9436D789A6A2D90]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCleanupOldIndexDirectories
   [junit4]   2> 3493998 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[B9436D789A6A2D90]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLocalityReporter
   [junit4]   2> 3494006 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53614/solr
   [junit4]   2> 3494006 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3494007 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53614/solr3/
   [junit4]   2> 3494017 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[B9436D789A6A2D90]) [    ] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:53614/solr3 for locality metrics.
   [junit4]   2> 3494027 INFO  (IPC Server handler 5 on 53614) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49771 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-8e976fbd-169e-45e7-8c0a-66c0a1b7d8ec:NORMAL:127.0.0.1:60123|RBW], ReplicaUC[[DISK]DS-f388a628-6c5e-42f1-a941-c613ebf40910:NORMAL:127.0.0.1:49771|FINALIZED]]} size 0
   [junit4]   2> 3494028 INFO  (IPC Server handler 9 on 53614) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60123 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f388a628-6c5e-42f1-a941-c613ebf40910:NORMAL:127.0.0.1:49771|FINALIZED], ReplicaUC[[DISK]DS-00975e1f-8675-4514-87c8-5a37214e0973:NORMAL:127.0.0.1:60123|FINALIZED]]} size 0
   [junit4]   2> 3494037 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[B9436D789A6A2D90]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLocalityReporter
   [junit4]   2> 3494068 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testInitArgsOrSysPropConfig
   [junit4]   2> 3494068 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53614/solr1
   [junit4]   2> 3494068 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3494069 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53614/solr2
   [junit4]   2> 3494069 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3494069 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53614/solr2
   [junit4]   2> 3494069 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3494070 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53614/solr
   [junit4]   2> 3494080 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3494080 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
   [junit4]   2> 3494080 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3495423 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[B9436D789A6A2D90]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testInitArgsOrSysPropConfig
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsDirectoryFactoryTest -Dtests.method=testInitArgsOrSysPropConfig -Dtests.seed=B9436D789A6A2D90 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=mt-MT -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1.39s J0 | HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([B9436D789A6A2D90:4EECA45367E3C7BB]:0)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Direct buffer memory
   [junit4]    > at java.nio.Bits.reserveMemory(Bits.java:693)
   [junit4]    > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
   [junit4]    > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
   [junit4]    > at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
   [junit4]    > ... 42 more
   [junit4]   2> 3495431 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 3495440 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3495543 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:53614) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-1676003404-10.41.0.5-1494617365198 (Datanode Uuid d2168978-e0c3-471d-8dce-6dd54416f792) service to localhost/127.0.0.1:53614 interrupted
   [junit4]   2> 3495543 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:53614) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1676003404-10.41.0.5-1494617365198 (Datanode Uuid d2168978-e0c3-471d-8dce-6dd54416f792) service to localhost/127.0.0.1:53614
   [junit4]   2> 3495547 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 3495560 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3495662 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:53614) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-1676003404-10.41.0.5-1494617365198 (Datanode Uuid 38f8f73f-72a3-4191-ba24-b21e75984763) service to localhost/127.0.0.1:53614 interrupted
   [junit4]   2> 3495662 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:53614) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1676003404-10.41.0.5-1494617365198 (Datanode Uuid 38f8f73f-72a3-4191-ba24-b21e75984763) service to localhost/127.0.0.1:53614
   [junit4]   2> 3495673 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3495775 ERROR (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.core.HdfsDirectoryFactoryTest.teardownClass(HdfsDirectoryFactoryTest.java:64)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> 3495779 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[B9436D789A6A2D90]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.HdfsDirectoryFactoryTest_B9436D789A6A2D90-001
   [junit4]   2> May 12, 2017 7:29:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true): {}, locale=mt-MT, timezone=Asia/Jayapura
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=347037568,total=519569408
   [junit4]   2> NOTE: All tests run in this JVM: [DistribJoinFromCollectionTest, TestCorePropertiesReload, StressHdfsTest, TestStressLucene, TestSolrQueryParserDefaultOperatorResource, BJQParserTest, OverseerRolesTest, TestLMJelinekMercerSimilarityFactory, TestBlobHandler, SignatureUpdateProcessorFactoryTest, ReturnFieldsTest, DirectoryFactoryTest, TestRandomDVFaceting, SuggestComponentTest, TestSolr4Spatial, PeerSyncTest, PeerSyncReplicationTest, TestSolrCoreProperties, TestCrossCoreJoin, TestRequestStatusCollectionAPI, TestCloudPseudoReturnFields, BigEndianAscendingWordDeserializerTest, TestCloudInspectUtil, TestTolerantUpdateProcessorRandomCloud, TestPayloadCheckQParserPlugin, VMParamsZkACLAndCredentialsProvidersTest, SolrMetricsIntegrationTest, CurrencyFieldOpenExchangeTest, DocValuesMissingTest, CoreAdminCreateDiscoverTest, SpellingQueryConverterTest, TestFunctionQuery, TestBadConfig, PreAnalyzedFieldManagedSchemaCloudTest, TestRangeQuery, CursorPagingTest, TestDistribDocBasedVersion, OverriddenZkACLAndCredentialsProvidersTest, TestAnalyzeInfixSuggestions, ConfigSetsAPITest, TestInPlaceUpdatesDistrib, VersionInfoTest, TolerantUpdateProcessorTest, ExplicitHLLTest, TestConfigSetImmutable, TestRecoveryHdfs, SolrJmxReporterTest, TestCollectionAPI, TestShortCircuitedRequests, HdfsLockFactoryTest, TestMinMaxOnMultiValuedField, TestSimpleTextCodec, CollectionReloadTest, TestReplicationHandlerBackup, ReplaceNodeTest, HLLUtilTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, MinimalSchemaTest, TestPseudoReturnFields, PreAnalyzedFieldTest, CopyFieldTest, PathHierarchyTokenizerFactoryTest, TestRemoteStreaming, HdfsTlogReplayBufferedWhileIndexingTest, TestValueSourceCache, TestCollapseQParserPlugin, TestRuleBasedAuthorizationPlugin, BlockDirectoryTest, TestMultiWordSynonyms, TestHighlightDedupGrouping, TestRawResponseWriter, URLClassifyProcessorTest, TestHdfsCloudBackupRestore, MissingSegmentRecoveryTest, BaseCdcrDistributedZkTest, TestJsonRequest, LeaderElectionContextKeyTest, StandardRequestHandlerTest, NumericFieldsTest, AnalysisAfterCoreReloadTest, TestRandomFaceting, MultiTermTest, TestJoin, TestOmitPositions, TestClassicSimilarityFactory, TestSchemaSimilarityResource, InfoHandlerTest, TestPerFieldSimilarityWithDefaultOverride, TestRealTimeGet, TestUseDocValuesAsStored2, ShardSplitTest, UUIDFieldTest, TestReplicationHandler, AddBlockUpdateTest, DistributedQueryComponentCustomSortTest, CollectionsAPIAsyncDistributedZkTest, TestCustomDocTransformer, TestImpersonationWithHadoopAuth, SolrShardReporterTest, TestDocBasedVersionConstraints, SolrXmlInZkTest, TestFastWriter, CursorMarkTest, TestChildDocTransformer, TestCustomSort, TestManagedSchemaAPI, NotRequiredUniqueKeyTest, DeleteNodeTest, TestDistributedSearch, TimeZoneUtilsTest, HighlighterMaxOffsetTest, RulesTest, SyncSliceTest, CustomCollectionTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, RAMDirectoryFactoryTest, TestLRUCache, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, FileUtilsTest, DistributedMLTComponentTest, TestRTGBase, DistributedIntervalFacetingTest, CdcrRequestHandlerTest, CleanupOldIndexTest, CreateCollectionCleanupTest, DeleteReplicaTest, DistributedQueueTest, DocValuesNotIndexedTest, LeaderInitiatedRecoveryOnCommitTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerModifyCollectionTest, RecoveryAfterSoftCommitTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, TestCloudDeleteByQuery, TestCloudRecovery, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestMiniSolrCloudCluster, TestRandomFlRTGCloud, TestSolrCloudWithKerberosAlt, TriLevelCompositeIdRoutingTest, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, HdfsSyncSliceTest, HdfsThreadLeakTest, HdfsWriteToMultipleCollectionsTest, TestClusterStateMutator, ZkStateWriterTest, CachingDirectoryFactoryTest, ConfigureRecoveryStrategyTest, HdfsDirectoryFactoryTest]
   [junit4] Completed [551/710 (1!)] on J0 in 13.35s, 3 tests, 1 error <<< FAILURES!

[...truncated 43863 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:655: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:643: Source checkout is modified!!! Offending files:
* solr/licenses/hsqldb-2.4.0.jar.sha1

Total time: 276 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1299 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1299/

1 tests failed.
FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([F4A63A2B6FAE3DAC:309F3009227D787]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more




Build Log:
[...truncated 13363 lines...]
   [junit4] Suite: org.apache.solr.core.HdfsDirectoryFactoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/init-core-data-001
   [junit4]   2> 3507237 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 3507237 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 3507240 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3507292 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3507300 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3507302 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3507316 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_45963_hdfs____xqq62n/webapp
   [junit4]   2> 3507654 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45963
   [junit4]   2> 3507729 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3507731 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3507744 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36670_datanode____v5wzlf/webapp
   [junit4]   2> 3508079 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36670
   [junit4]   2> 3508129 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3508130 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3508144 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46199_datanode____ajxbxy/webapp
   [junit4]   2> 3508173 INFO  (IPC Server handler 7 on 45764) [    ] BlockStateChange BLOCK* processReport: from storage DS-be62be90-e23e-4152-b731-9a16e401dd52 node DatanodeRegistration(127.0.0.1:46451, datanodeUuid=1ca1d35e-8196-4f0a-9afa-09d8c7d79f51, infoPort=60747, infoSecurePort=0, ipcPort=39017, storageInfo=lv=-56;cid=testClusterID;nsid=77805172;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3508173 INFO  (IPC Server handler 7 on 45764) [    ] BlockStateChange BLOCK* processReport: from storage DS-9ee439b1-5f0f-4d7c-acb2-2539a53cbc02 node DatanodeRegistration(127.0.0.1:46451, datanodeUuid=1ca1d35e-8196-4f0a-9afa-09d8c7d79f51, infoPort=60747, infoSecurePort=0, ipcPort=39017, storageInfo=lv=-56;cid=testClusterID;nsid=77805172;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3508536 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46199
   [junit4]   2> 3508644 INFO  (IPC Server handler 5 on 45764) [    ] BlockStateChange BLOCK* processReport: from storage DS-359fae32-d6ca-4061-9a26-4b68a3bb014f node DatanodeRegistration(127.0.0.1:46313, datanodeUuid=14ee64d7-8da3-4d35-8a37-0924d7e62663, infoPort=39970, infoSecurePort=0, ipcPort=38798, storageInfo=lv=-56;cid=testClusterID;nsid=77805172;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3508644 INFO  (IPC Server handler 5 on 45764) [    ] BlockStateChange BLOCK* processReport: from storage DS-8e5a2e95-739a-4a35-93c6-119f35601c49 node DatanodeRegistration(127.0.0.1:46313, datanodeUuid=14ee64d7-8da3-4d35-8a37-0924d7e62663, infoPort=39970, infoSecurePort=0, ipcPort=38798, storageInfo=lv=-56;cid=testClusterID;nsid=77805172;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3508710 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLocalityReporter
   [junit4]   2> 3508718 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45764/solr
   [junit4]   2> 3508719 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3508719 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45764/solr3/
   [junit4]   2> 3508730 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:45764/solr3 for locality metrics.
   [junit4]   2> 3508739 INFO  (IPC Server handler 5 on 45764) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46313 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9ee439b1-5f0f-4d7c-acb2-2539a53cbc02:NORMAL:127.0.0.1:46451|RBW], ReplicaUC[[DISK]DS-359fae32-d6ca-4061-9a26-4b68a3bb014f:NORMAL:127.0.0.1:46313|FINALIZED]]} size 0
   [junit4]   2> 3508741 INFO  (IPC Server handler 2 on 45764) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46451 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9ee439b1-5f0f-4d7c-acb2-2539a53cbc02:NORMAL:127.0.0.1:46451|RBW], ReplicaUC[[DISK]DS-359fae32-d6ca-4061-9a26-4b68a3bb014f:NORMAL:127.0.0.1:46313|FINALIZED]]} size 0
   [junit4]   2> 3508746 INFO  (TEST-HdfsDirectoryFactoryTest.testLocalityReporter-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLocalityReporter
   [junit4]   2> 3508780 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testInitArgsOrSysPropConfig
   [junit4]   2> 3508780 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45764/solr1
   [junit4]   2> 3508781 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3508781 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45764/solr2
   [junit4]   2> 3508781 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3508781 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45764/solr2
   [junit4]   2> 3508781 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3508782 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45764/solr
   [junit4]   2> 3508791 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3508791 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [134217728] will allocate [1] slabs and use ~[134217728] bytes
   [junit4]   2> 3508792 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3510021 INFO  (TEST-HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testInitArgsOrSysPropConfig
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsDirectoryFactoryTest -Dtests.method=testInitArgsOrSysPropConfig -Dtests.seed=F4A63A2B6FAE3DAC -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=sr-ME -Dtests.timezone=IET -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   1.28s J2 | HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F4A63A2B6FAE3DAC:309F3009227D787]:0)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Direct buffer memory
   [junit4]    > at java.nio.Bits.reserveMemory(Bits.java:693)
   [junit4]    > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
   [junit4]    > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
   [junit4]    > at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
   [junit4]    > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
   [junit4]    > ... 42 more
   [junit4]   2> 3510062 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCleanupOldIndexDirectories
   [junit4]   2> 3510062 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45764/solr1
   [junit4]   2> 3510062 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3510076 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Found 1 old index directories to clean-up under hdfs://localhost:45764/solr1/mock/data afterReload=false
   [junit4]   2> 3510077 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.c.HdfsDirectoryFactory Deleted old index directory hdfs://localhost:45764/solr1/mock/data/index.20170513170527954
   [junit4]   2> 3510078 INFO  (TEST-HdfsDirectoryFactoryTest.testCleanupOldIndexDirectories-seed#[F4A63A2B6FAE3DAC]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCleanupOldIndexDirectories
   [junit4]   2> 3510080 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 3510089 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3510190 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:45764) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-796841399-10.41.0.5-1494709525156 (Datanode Uuid 14ee64d7-8da3-4d35-8a37-0924d7e62663) service to localhost/127.0.0.1:45764 interrupted
   [junit4]   2> 3510190 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:45764) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-796841399-10.41.0.5-1494709525156 (Datanode Uuid 14ee64d7-8da3-4d35-8a37-0924d7e62663) service to localhost/127.0.0.1:45764
   [junit4]   2> 3510193 WARN  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 3510203 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3510305 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:45764) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-796841399-10.41.0.5-1494709525156 (Datanode Uuid 1ca1d35e-8196-4f0a-9afa-09d8c7d79f51) service to localhost/127.0.0.1:45764 interrupted
   [junit4]   2> 3510305 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:45764) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-796841399-10.41.0.5-1494709525156 (Datanode Uuid 1ca1d35e-8196-4f0a-9afa-09d8c7d79f51) service to localhost/127.0.0.1:45764
   [junit4]   2> 3510315 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 3510417 ERROR (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.core.HdfsDirectoryFactoryTest.teardownClass(HdfsDirectoryFactoryTest.java:64)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> 3510421 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[F4A63A2B6FAE3DAC]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_F4A63A2B6FAE3DAC-001
   [junit4]   2> May 13, 2017 9:05:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1545, maxMBSortInHeap=6.39147047743192, sim=RandomSimilarity(queryNorm=false): {}, locale=sr-ME, timezone=IET
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=284625992,total=473432064
   [junit4]   2> NOTE: All tests run in this JVM: [FieldMutatingUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestRandomRequestDistribution, TestBulkSchemaConcurrent, SharedFSAutoReplicaFailoverUtilsTest, MetricsConfigTest, TestRecovery, TestIndexSearcher, DistanceUnitsTest, UnloadDistributedZkTest, PropertiesRequestHandlerTest, TestZkChroot, TestConfigsApi, HttpSolrCallGetCoreTest, UpdateLogTest, TestUseDocValuesAsStored, SolrTestCaseJ4Test, TestPKIAuthenticationPlugin, TestBinaryResponseWriter, TestBinaryField, CdcrVersionReplicationTest, TestNamedUpdateProcessors, HdfsBasicDistributedZkTest, DateFieldTest, TestCloudSchemaless, TestSearcherReuse, HighlighterConfigTest, TestNonDefinedSimilarityFactory, TestIBSimilarityFactory, TestExpandComponent, TestDistributedMissingSort, SuggesterTSTTest, TestSlowCompositeReaderWrapper, BlockJoinFacetRandomTest, OutputWriterTest, PrimitiveFieldTypeTest, TestTolerantUpdateProcessorCloud, TestTolerantSearch, MBeansHandlerTest, SimpleFacetsTest, HardAutoCommitTest, SpellCheckCollatorWithCollapseTest, RequestHandlersTest, TestSolrConfigHandlerCloud, TestCloudNestedDocsSort, ClassificationUpdateProcessorIntegrationTest, ManagedSchemaRoundRobinCloudTest, HLLSerializationTest, LeaderFailureAfterFreshStartTest, TestFileDictionaryLookup, TestExtendedDismaxParser, TestExactStatsCache, TestLeaderElectionZkExpiry, TestFastOutputStream, BinaryUpdateRequestHandlerTest, TestXmlQParser, SuggestComponentContextFilterQueryTest, PreAnalyzedUpdateProcessorTest, ReplicaListTransformerTest, UpdateParamsTest, TestPHPSerializedResponseWriter, TestMergePolicyConfig, TestLMDirichletSimilarityFactory, TestSerializedLuceneMatchVersion, SynonymTokenizerTest, LeaderElectionTest, DirectUpdateHandlerOptimizeTest, RemoteQueryErrorTest, OverseerTest, RequestLoggingTest, TestFieldTypeCollectionResource, TestQuerySenderNoQuery, DistributedSuggestComponentTest, TestUniqueKeyFieldResource, PluginInfoTest, TestFieldCache, CollectionTooManyReplicasTest, CurrencyFieldXmlFileTest, MergeStrategyTest, FieldAnalysisRequestHandlerTest, TestShardHandlerFactory, TestManagedSchema, TestCloudDeleteByQuery, CheckHdfsIndexTest, DistributedFacetPivotWhiteBoxTest, TestCoreDiscovery, V2ApiIntegrationTest, AutoCommitTest, TestEmbeddedSolrServerSchemaAPI, DistributedExpandComponentTest, TestSolrXml, JavabinLoaderTest, TestReload, TestSolr4Spatial2, EchoParamsTest, ExternalFileFieldSortTest, TestCloudPivotFacet, TestPushWriter, CoreMergeIndexesAdminHandlerTest, BadIndexSchemaTest, TestConfigSets, TestCSVResponseWriter, OverseerModifyCollectionTest, TestSubQueryTransformerCrossCore, TestStandardQParsers, DirectUpdateHandlerTest, TestScoreJoinQPScore, SuggesterWFSTTest, SimpleMLTQParserTest, TestRTimerTree, TestMacros, BitVectorTest, TestMissingGroups, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, ZkSolrClientTest, TestRandomDVFaceting, TestDistributedGrouping, TestFaceting, TestHashPartitioner, TestStressReorder, TestSolr4Spatial, StatsComponentTest, TestSort, TestFunctionQuery, HighlighterTest, ShowFileRequestHandlerTest, SpellCheckCollatorTest, SolrRequestParserTest, SuggesterTest, TestTrie, SchemaVersionSpecificBehaviorTest, TestAtomicUpdateErrorCases, DocValuesMultiTest, TestSolrDeletionPolicy1, DebugComponentTest, LukeRequestHandlerTest, TestReversedWildcardFilterFactory, TestOmitPositions, XmlUpdateRequestHandlerTest, TermVectorComponentTest, TestSolrQueryParser, SolrPluginUtilsTest, TestCollationField, UpdateRequestProcessorFactoryTest, CSVRequestHandlerTest, TestComponentsName, TestQuerySenderListener, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestJettySolrRunner, ActionThrottleTest, AliasIntegrationTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionsAPIAsyncDistributedZkTest, ConfigSetsAPITest, CustomCollectionTest, DeleteNodeTest, DistribCursorPagingTest, DocValuesNotIndexedTest, ForceLeaderTest, OverseerCollectionConfigSetProcessorTest, OverseerTaskQueueTest, RestartWhileUpdatingTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudInspectUtil, TestCloudPseudoReturnFields, TestClusterProperties, TestCollectionAPI, TestConfigSetsAPIExclusivity, TestDistribDocBasedVersion, TestLeaderElectionWithEmptyReplica, TestLocalFSCloudBackupRestore, TestMiniSolrCloudClusterSSL, TestRandomFlRTGCloud, TlogReplayBufferedWhileIndexingTest, HdfsNNFailoverTest, HdfsRecoveryZkTest, HdfsThreadLeakTest, HdfsUnloadDistributedZkTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest, ZkStateWriterTest, RuleEngineTest, RulesTest, CoreSorterTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest]
   [junit4] Completed [568/710 (1!)] on J2 in 12.88s, 3 tests, 1 error <<< FAILURES!

[...truncated 43824 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1300 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1300/

6 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=34388, name=Connection evictor, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=34388, name=Connection evictor, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsNNFailoverTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:361)  at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3015)  at org.apache.solr.core.SolrCore.close(SolrCore.java:1579)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:994)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:855)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:960)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:595)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [NRTCachingDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:361)
        at org.apache.solr.core.SolrCore.cleanupOldIndexDirectories(SolrCore.java:3015)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1579)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:994)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:855)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:960)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:595)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


        at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C:1AEA54E77DF740B2]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.junit.Assert.assertNotNull(Assert.java:537)
        at org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter(MetricsHandlerTest.java:201)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
        at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C:8F28638D4FF0CAA7]:0)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
        at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:293)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
        at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
        at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
        ... 42 more




Build Log:
[...truncated 11655 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.MetricsHandlerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.MetricsHandlerTest_7887AAA6B279208C-001/init-core-data-001
   [junit4]   2> 184672 WARN  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 184673 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 184674 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 184674 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 184675 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 184712 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 184736 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 184876 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 184935 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2604c2
   [junit4]   2> 184939 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2604c2
   [junit4]   2> 184940 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2604c2
   [junit4]   2> 184941 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 184977 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 185000 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 185104 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 185112 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 185112 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2604c2
   [junit4]   2> 185112 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 185112 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.MetricsHandlerTest_7887AAA6B279208C-001/init-core-data-001/]
   [junit4]   2> 185114 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=24, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 185183 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 185183 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 185184 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 185184 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 185185 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=51.8056640625, floorSegmentMB=1.5537109375, forceMergeDeletesPctAllowed=11.89262132254437, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8098542025281875
   [junit4]   2> 185186 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@15d33bbf[collection1] main]
   [junit4]   2> 185187 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 185193 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 185220 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 185220 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 185220 INFO  (coreLoadExecutor-515-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567404477015654400
   [junit4]   2> 185221 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 185221 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 185221 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 185224 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 185227 INFO  (TEST-MetricsHandlerTest.testPropertyFilter-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPropertyFilter
   [junit4]   2> 185228 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 185228 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 185236 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 185236 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 185236 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 185238 INFO  (searcherExecutor-516-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@15d33bbf[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 185247 INFO  (TEST-MetricsHandlerTest.testPropertyFilter-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPropertyFilter
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MetricsHandlerTest -Dtests.method=testPropertyFilter -Dtests.seed=7887AAA6B279208C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=hr-HR -Dtests.timezone=America/Indiana/Knox -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.03s J1 | MetricsHandlerTest.testPropertyFilter <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C:1AEA54E77DF740B2]:0)
   [junit4]    > at org.apache.solr.handler.admin.MetricsHandlerTest.testPropertyFilter(MetricsHandlerTest.java:201)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 185251 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCompact
   [junit4]   2> 185263 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[7887AAA6B279208C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 185278 INFO  (TEST-MetricsHandlerTest.testCompact-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCompact
   [junit4]   2> 185280 INFO  (TEST-MetricsHandlerTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 185363 INFO  (TEST-MetricsHandlerTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> 185364 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 185364 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2146240490
   [junit4]   2> 185364 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 185365 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 185367 INFO  (SUITE-MetricsHandlerTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 185381 INFO  (coreCloseExecutor-521-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@ac81dbe
   [junit4]   2> 185397 INFO  (coreCloseExecutor-521-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=180886974
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.MetricsHandlerTest_7887AAA6B279208C-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=hr-HR, timezone=America/Indiana/Knox
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=123888992,total=395837440
   [junit4]   2> NOTE: All tests run in this JVM: [AlternateDirectoryTest, TestFuzzyAnalyzedSuggestions, ChangedSchemaMergeTest, SSLMigrationTest, TestSmileRequest, TestHalfAndHalfDocValues, TestTrieFacet, TestCursorMarkWithoutUniqueKey, PeerSyncWithIndexFingerprintCachingTest, TestNumericTerms64, DeleteInactiveReplicaTest, TestTestInjection, ExitableDirectoryReaderTest, OverseerCollectionConfigSetProcessorTest, TestSchemaNameResource, TestOrdValues, TestSweetSpotSimilarityFactory, TestSha256AuthenticationProvider, DocumentBuilderTest, AssignTest, CoreAdminRequestStatusTest, DistributedIntervalFacetingTest, DistributedFacetPivotSmallTest, MetricsHandlerTest]
   [junit4] Completed [42/710 (1!)] on J1 in 0.75s, 3 tests, 1 failure <<< FAILURES!

[...truncated 1729 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsNNFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/init-core-data-001
   [junit4]   2> 5207883 WARN  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1187 numCloses=1187
   [junit4]   2> 5207884 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 5207885 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 5207885 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 5207972 WARN  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 5207995 WARN  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 5208000 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 5208020 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_59147_hdfs____k1n94q/webapp
   [junit4]   2> 5208438 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59147
   [junit4]   2> 5208563 WARN  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 5208565 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 5208579 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_43522_datanode____f0gmyl/webapp
   [junit4]   2> 5208991 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43522
   [junit4]   2> 5209066 WARN  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 5209068 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 5209082 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_55748_datanode____oja8jm/webapp
   [junit4]   2> 5209213 INFO  (IPC Server handler 5 on 53086) [    ] BlockStateChange BLOCK* processReport: from storage DS-8fc6919e-e8e4-41f2-b4cf-f00c32000ff6 node DatanodeRegistration(127.0.0.1:60027, datanodeUuid=6d6fee6d-af55-4df0-8ced-aba5f918dca0, infoPort=46220, infoSecurePort=0, ipcPort=52877, storageInfo=lv=-56;cid=testClusterID;nsid=1225727325;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 5209213 INFO  (IPC Server handler 5 on 53086) [    ] BlockStateChange BLOCK* processReport: from storage DS-fd9b4832-165f-42c5-ad44-5cdf7b06d16a node DatanodeRegistration(127.0.0.1:60027, datanodeUuid=6d6fee6d-af55-4df0-8ced-aba5f918dca0, infoPort=46220, infoSecurePort=0, ipcPort=52877, storageInfo=lv=-56;cid=testClusterID;nsid=1225727325;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 5209514 INFO  (SUITE-HdfsNNFailoverTest-seed#[7887AAA6B279208C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55748
   [junit4]   2> 5209624 INFO  (IPC Server handler 1 on 53086) [    ] BlockStateChange BLOCK* processReport: from storage DS-2c2af4b2-3adb-49fe-abbd-726233f425ca node DatanodeRegistration(127.0.0.1:45912, datanodeUuid=ee622b01-8f94-44e6-8ce4-435f05fdfd51, infoPort=32884, infoSecurePort=0, ipcPort=38689, storageInfo=lv=-56;cid=testClusterID;nsid=1225727325;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 5209624 INFO  (IPC Server handler 1 on 53086) [    ] BlockStateChange BLOCK* processReport: from storage DS-046e84c4-f465-42de-84d8-6c05f5905df0 node DatanodeRegistration(127.0.0.1:45912, datanodeUuid=ee622b01-8f94-44e6-8ce4-435f05fdfd51, infoPort=32884, infoSecurePort=0, ipcPort=38689, storageInfo=lv=-56;cid=testClusterID;nsid=1225727325;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 5209862 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 5209862 INFO  (Thread-75122) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 5209862 INFO  (Thread-75122) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 5209865 ERROR (Thread-75122) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 5209962 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.ZkTestServer start zk server on port:49154
   [junit4]   2> 5209984 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 5209990 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 5209993 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 5209995 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 5209997 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 5209998 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 5210000 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 5210002 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 5210003 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 5210005 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 5210006 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 5210139 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/control-001/cores/collection1
   [junit4]   2> 5210141 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 5210144 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4294957b{/,null,AVAILABLE}
   [junit4]   2> 5210144 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1b8a1cb7{HTTP/1.1,[http/1.1]}{127.0.0.1:60927}
   [junit4]   2> 5210146 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.Server Started @5213060ms
   [junit4]   2> 5210146 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:53086/hdfs__localhost_53086__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001_tempDir-002_control_data, hostContext=/, hostPort=60927, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/control-001/cores}
   [junit4]   2> 5210147 ERROR (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 5210147 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 5210147 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 5210147 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 5210147 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-14T21:46:57.843Z
   [junit4]   2> 5210150 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 5210151 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/control-001/solr.xml
   [junit4]   2> 5210157 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 5210161 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49154/solr
   [junit4]   2> 5210186 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5210187 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60927_
   [junit4]   2> 5210188 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.c.Overseer Overseer (id=97963109093212164-127.0.0.1:60927_-n_0000000000) starting
   [junit4]   2> 5210203 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60927_
   [junit4]   2> 5210206 INFO  (zkCallback-34161-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 5210307 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5210313 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5210313 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5210315 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/control-001/cores
   [junit4]   2> 5210315 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:60927_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 5210317 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 5210319 INFO  (OverseerStateUpdate-97963109093212164-127.0.0.1:60927_-n_0000000000) [n:127.0.0.1:60927_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 5211342 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 5211361 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 5211530 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 5211544 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 5211545 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.core_node1' (registry 'solr.core.control_collection.shard1.core_node1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5211545 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53086/solr_hdfs_home
   [junit4]   2> 5211545 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 5211545 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 5211545 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/control-001/cores/collection1], dataDir=[hdfs://localhost:53086/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 5211547 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 5211556 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 5211557 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 5211557 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 5211570 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 5211571 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 5211596 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 5211606 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 5211606 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 5211606 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 5211616 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 5211616 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7991362844850747]
   [junit4]   2> 5211638 INFO  (IPC Server handler 3 on 53086) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60027 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2c2af4b2-3adb-49fe-abbd-726233f425ca:NORMAL:127.0.0.1:45912|RBW], ReplicaUC[[DISK]DS-fd9b4832-165f-42c5-ad44-5cdf7b06d16a:NORMAL:127.0.0.1:60027|RBW]]} size 0
   [junit4]   2> 5211639 INFO  (IPC Server handler 9 on 53086) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45912 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2c2af4b2-3adb-49fe-abbd-726233f425ca:NORMAL:127.0.0.1:45912|RBW], ReplicaUC[[DISK]DS-fd9b4832-165f-42c5-ad44-5cdf7b06d16a:NORMAL:127.0.0.1:60027|RBW]]} size 0
   [junit4]   2> 5211645 WARN  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 5211704 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 5211704 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 5211704 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 5211728 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5211729 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5211730 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3b049f9e
   [junit4]   2> 5211735 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c5edd6e[collection1] main]
   [junit4]   2> 5211737 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 5211737 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 5211738 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5211739 INFO  (searcherExecutor-13834-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@c5edd6e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5211739 INFO  (coreLoadExecutor-13833-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567409747457474560
   [junit4]   2> 5211747 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 5211747 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 5211747 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60927/collection1/
   [junit4]   2> 5211747 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 5211747 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:60927/collection1/ has no replicas
   [junit4]   2> 5211747 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 5211751 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60927/collection1/ shard1
   [junit4]   2> 5211839 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 5211840 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49154/solr ready
   [junit4]   2> 5211840 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 5211840 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 5211852 INFO  (coreZkRegister-13826-thread-1-processing-n:127.0.0.1:60927_ x:collection1 c:control_collection) [n:127.0.0.1:60927_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 5211973 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-1-001/cores/collection1
   [junit4]   2> 5211973 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-1-001
   [junit4]   2> 5211974 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 5211976 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64f8545{/,null,AVAILABLE}
   [junit4]   2> 5211977 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5ad0b557{HTTP/1.1,[http/1.1]}{127.0.0.1:51926}
   [junit4]   2> 5211979 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.Server Started @5214893ms
   [junit4]   2> 5211979 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:53086/hdfs__localhost_53086__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=51926, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-1-001/cores}
   [junit4]   2> 5211983 ERROR (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 5211985 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 5211986 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 5211986 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 5211986 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-14T21:46:59.682Z
   [junit4]   2> 5211990 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 5211990 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-1-001/solr.xml
   [junit4]   2> 5211996 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 5211999 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49154/solr
   [junit4]   2> 5212017 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 5212022 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5212026 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51926_
   [junit4]   2> 5212028 INFO  (zkCallback-34161-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 5212030 INFO  (zkCallback-34165-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 5212035 INFO  (zkCallback-34171-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 5212133 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5212140 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5212140 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5212142 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-1-001/cores
   [junit4]   2> 5212143 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:51926_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 5212147 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 5212151 INFO  (OverseerStateUpdate-97963109093212164-127.0.0.1:60927_-n_0000000000) [n:127.0.0.1:60927_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 5212253 INFO  (zkCallback-34171-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 5213168 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 5213184 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 5213321 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 5213332 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 5213333 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.core_node1' (registry 'solr.core.collection1.shard1.core_node1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5213333 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53086/solr_hdfs_home
   [junit4]   2> 5213333 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 5213333 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 5213333 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:53086/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 5213334 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 5213343 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 5213343 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 5213343 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 5213347 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 5213348 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 5213365 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 5213372 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 5213372 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 5213372 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 5213376 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 5213376 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7991362844850747]
   [junit4]   2> 5213391 INFO  (IPC Server handler 5 on 53086) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45912 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-fd9b4832-165f-42c5-ad44-5cdf7b06d16a:NORMAL:127.0.0.1:60027|RBW], ReplicaUC[[DISK]DS-046e84c4-f465-42de-84d8-6c05f5905df0:NORMAL:127.0.0.1:45912|RBW]]} size 0
   [junit4]   2> 5213392 INFO  (IPC Server handler 1 on 53086) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60027 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-046e84c4-f465-42de-84d8-6c05f5905df0:NORMAL:127.0.0.1:45912|RBW], ReplicaUC[[DISK]DS-8fc6919e-e8e4-41f2-b4cf-f00c32000ff6:NORMAL:127.0.0.1:60027|FINALIZED]]} size 0
   [junit4]   2> 5213396 WARN  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 5213448 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 5213448 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 5213448 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 5213461 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5213462 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5213464 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@74deccf0
   [junit4]   2> 5213470 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7359ef42[collection1] main]
   [junit4]   2> 5213471 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 5213472 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 5213473 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5213474 INFO  (searcherExecutor-13845-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7359ef42[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5213474 INFO  (coreLoadExecutor-13844-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567409749276753920
   [junit4]   2> 5213484 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 5213484 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 5213484 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:51926/collection1/
   [junit4]   2> 5213484 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 5213484 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:51926/collection1/ has no replicas
   [junit4]   2> 5213484 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 5213488 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:51926/collection1/ shard1
   [junit4]   2> 5213592 INFO  (zkCallback-34171-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 5213639 INFO  (coreZkRegister-13839-thread-1-processing-n:127.0.0.1:51926_ x:collection1 c:collection1) [n:127.0.0.1:51926_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 5213743 INFO  (zkCallback-34171-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 5213781 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-2-001/cores/collection1
   [junit4]   2> 5213781 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-2-001
   [junit4]   2> 5213782 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 5213783 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15810aa9{/,null,AVAILABLE}
   [junit4]   2> 5213784 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a825359{HTTP/1.1,[http/1.1]}{127.0.0.1:34295}
   [junit4]   2> 5213785 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.e.j.s.Server Started @5216699ms
   [junit4]   2> 5213785 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:53086/hdfs__localhost_53086__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=34295, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-2-001/cores}
   [junit4]   2> 5213786 ERROR (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 5213787 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 5213787 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 5213787 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 5213787 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-14T21:47:01.483Z
   [junit4]   2> 5213790 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 5213790 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-2-001/solr.xml
   [junit4]   2> 5213795 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 5213798 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49154/solr
   [junit4]   2> 5213807 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 5213809 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5213812 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34295_
   [junit4]   2> 5213813 INFO  (zkCallback-34171-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5213813 INFO  (zkCallback-34161-thread-1-processing-n:127.0.0.1:60927_) [n:127.0.0.1:60927_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5213813 INFO  (zkCallback-34165-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5213816 INFO  (zkCallback-34177-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5213941 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5213947 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5213947 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5213949 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-2-001/cores
   [junit4]   2> 5213950 INFO  (TEST-HdfsNNFailoverTest.test-seed#[7887AAA6B279208C]) [n:127.0.0.1:34295_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 5213954 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 5213956 INFO  (OverseerStateUpdate-97963109093212164-127.0.0.1:60927_-n_0000000000) [n:127.0.0.1:60927_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 5214059 INFO  (zkCallback-34171-thread-1-processing-n:127.0.0.1:51926_) [n:127.0.0.1:51926_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 5214059 INFO  (zkCallback-34177-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 5214972 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 5214991 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 5215175 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 5215189 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 5215189 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.core_node2' (registry 'solr.core.collection1.shard1.core_node2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c111bdc
   [junit4]   2> 5215190 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53086/solr_hdfs_home
   [junit4]   2> 5215190 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 5215190 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 5215190 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsNNFailoverTest_7887AAA6B279208C-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:53086/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 5215191 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 5215201 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 5215202 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 5215202 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 5215205 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 5215206 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 5215223 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:53086/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 5215230 INFO  (coreLoadExecutor-13855-thread-1-processing-n:127.0.0.1:34295_) [n:127.0.0.1:34295_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 5215230 INFO  (coreLoadExecutor-13855-th

[...truncated too long message...]

2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 52parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 42parent_f2_s=PostingsFormat(name=Memory), 32parent_f2_s=PostingsFormat(name=Direct), 31parent_f1_s=PostingsFormat(name=Direct), 5grand1_s=PostingsFormat(name=Direct), 6grand4_s=PostingsFormat(name=Direct), 11parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 51parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 54parent_f1_s=BlockTreeOrds(blocksize=128), 21parent_f1_s=BlockTreeOrds(blocksize=128), 4grand2_s=PostingsFormat(name=Direct), 7grand3_s=PostingsFormat(name=Direct), 41parent_f1_s=PostingsFormat(name=Memory), 37parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 40parent_f2_s=BlockTreeOrds(blocksize=128), 50parent_f2_s=PostingsFormat(name=Direct), 47parent_f1_s=BlockTreeOrds(blocksize=128), 14parent_f1_s=BlockTreeOrds(blocksize=128), 20parent_f2_s=PostingsFormat(name=Memory), 34parent_f1_s=PostingsFormat(name=Memory), 24parent_f1_s=PostingsFormat(name=Direct), 44parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 30parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 5parent_f1_s=BlockTreeOrds(blocksize=128), 6grand0_s=PostingsFormat(name=Direct), 46parent_f2_s=PostingsFormat(name=Memory), 10parent_f2_s=PostingsFormat(name=Direct), 8parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 27parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 2parent_f2_s=BlockTreeOrds(blocksize=128), 4child1_s=PostingsFormat(name=Memory), 2grand1_s=BlockTreeOrds(blocksize=128), 17parent_f2_s=BlockTreeOrds(blocksize=128), 3child2_s=PostingsFormat(name=Memory), 16parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 26parent_f2_s=BlockTreeOrds(blocksize=128), 1child0_s=PostingsFormat(name=Memory), 36parent_f2_s=PostingsFormat(name=Direct), 7grand6_s=PostingsFormat(name=Memory), 18parent_f1_s=PostingsFormat(name=Memory), 28parent_f1_s=PostingsFormat(name=Direct), 5grand0_s=PostingsFormat(name=Memory), 6grand5_s=PostingsFormat(name=Memory), 7grand2_s=PostingsFormat(name=Memory), 48parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 15parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 4grand3_s=PostingsFormat(name=Memory), 6parent_f1_s=PostingsFormat(name=Direct), 25parent_f1_s=BlockTreeOrds(blocksize=128), 0parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 3grand2_s=PostingsFormat(name=Memory), 35parent_f1_s=PostingsFormat(name=Direct), 19parent_f2_s=PostingsFormat(name=Memory), 4parent_f2_s=PostingsFormat(name=Memory), 37parent_f2_s=BlockTreeOrds(blocksize=128), 47parent_f2_s=PostingsFormat(name=Direct), 38parent_f1_s=PostingsFormat(name=Memory), 5grand4_s=PostingsFormat(name=Memory), 6grand1_s=PostingsFormat(name=Memory), 5parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 46parent_f1_s=PostingsFormat(name=Direct), 16parent_f1_s=BlockTreeOrds(blocksize=128), 8parent_f2_s=PostingsFormat(name=Memory), 39parent_f2_s=PostingsFormat(name=Memory), 29parent_f2_s=PostingsFormat(name=Direct), 2parent_f1_s=PostingsFormat(name=Direct), 4child2_s=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Memory), 27parent_f1_s=PostingsFormat(name=Memory), 45parent_f1_s=PostingsFormat(name=Memory), 17parent_f1_s=PostingsFormat(name=Direct), 2grand0_s=PostingsFormat(name=Direct), 49parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 3child1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 55parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 7parent_f2_s=PostingsFormat(name=Direct), 26parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 2child0_s=PostingsFormat(name=Direct), 3parent_f1_s=PostingsFormat(name=Memory), 36parent_f1_s=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1615, maxMBSortInHeap=7.963572630248015, sim=RandomSimilarity(queryNorm=false): {}, locale=es-DO, timezone=Australia/North
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=51,free=59144128,total=517996544
   [junit4]   2> NOTE: All tests run in this JVM: [AlternateDirectoryTest, TestFuzzyAnalyzedSuggestions, ChangedSchemaMergeTest, SSLMigrationTest, TestSmileRequest, TestHalfAndHalfDocValues, TestTrieFacet, TestCursorMarkWithoutUniqueKey, PeerSyncWithIndexFingerprintCachingTest, TestNumericTerms64, DeleteInactiveReplicaTest, TestTestInjection, ExitableDirectoryReaderTest, OverseerCollectionConfigSetProcessorTest, TestSchemaNameResource, TestOrdValues, TestSweetSpotSimilarityFactory, TestSha256AuthenticationProvider, DocumentBuilderTest, AssignTest, CoreAdminRequestStatusTest, DistributedIntervalFacetingTest, DistributedFacetPivotSmallTest, MetricsHandlerTest, TestSolrConfigHandler, BasicFunctionalityTest, SubstringBytesRefFilterTest, SolrMetricManagerTest, ConfigureRecoveryStrategyTest, TestLocalFSCloudBackupRestore, TestSolrQueryParserResource, AnalysisErrorHandlingTest, SegmentsInfoRequestHandlerTest, CdcrRequestHandlerTest, DistributedFacetPivotLargeTest, SpatialFilterTest, TestRandomFlRTGCloud, TestInfoStreamLogging, TestConfigSetsAPI, OverseerStatusTest, TestFiltering, TestTrie, ClassificationUpdateProcessorFactoryTest, JvmMetricsTest, QueryParsingTest, TestCloudJSONFacetJoinDomain, TestBlendedInfixSuggestions, ConnectionManagerTest, TestSolrIndexConfig, TestBM25SimilarityFactory, TestStressVersions, TestSchemalessBufferedUpdates, EnumFieldTest, InfixSuggestersTest, ZkControllerTest, TestElisionMultitermQuery, ZkCLITest, TestSolrFieldCacheBean, HighlighterTest, TestSolrCoreSnapshots, SharedFSAutoReplicaFailoverTest, TestUpdate, LoggingHandlerTest, DeleteReplicaTest, TestSubQueryTransformer, TestSSLRandomization, TestSizeLimitedDistributedMap, ConvertedLegacyTest, LargeFieldTest, TestHashPartitioner, SortByFunctionTest, SortSpecParsingTest, TestManagedSynonymFilterFactory, TestSchemaVersionResource, IndexSchemaTest, OnlyLeaderIndexesTest, TestCryptoKeys, PrimUtilsTest, TestFieldResource, ConjunctionSolrSpellCheckerTest, SolrCLIZkUtilsTest, AddSchemaFieldsUpdateProcessorFactoryTest, UpdateRequestProcessorFactoryTest, SliceStateTest, TestCharFilters, HdfsChaosMonkeySafeLeaderTest, TestNoOpRegenerator, DistributedMLTComponentTest, HdfsSyncSliceTest, TemplateUpdateProcessorTest, TestManagedResource, TestAuthorizationFramework, SystemInfoHandlerTest, BlobRepositoryCloudTest, SolrCmdDistributorTest, HdfsBasicDistributedZk2Test, CollectionsAPISolrJTest, GraphQueryTest, TestReloadDeadlock, BlockJoinFacetDistribTest, TestObjectReleaseTracker, TermsComponentTest, SolrInfoBeanTest, TestJavabinTupleStreamParser, ScriptEngineTest, DistribCursorPagingTest, CollectionStateFormat2Test, MoveReplicaTest, RecoveryAfterSoftCommitTest, DistributedDebugComponentTest, BasicAuthIntegrationTest, TestConfigReload, UUIDUpdateProcessorFallbackTest, SOLR749Test, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=7887AAA6B279208C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-DO -Dtests.timezone=Australia/North -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | FullSolrCloudDistribCmdsTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7887AAA6B279208C]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=34388, name=Connection evictor, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4] Completed [710/710 (4!)] on J1 in 7226.18s, 1 test, 3 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20170514_202004_7238128778113349719355.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid20719.hprof ...
   [junit4] Heap dump file created [627382907 bytes in 23.062 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 7457 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:727: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid20719.hprof

Total time: 373 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1301 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1301/

1 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1494891787746,generation=2,filelist=[_9i.cfe, _9i.cfs, _9i.si, _9n.fdt, _9n.fdx, _9n.fnm, _9n.nvd, _9n.nvm, _9n.si, _9n_TestBloomFilteredLucenePostings_0.blm, _9n_TestBloomFilteredLucenePostings_0.doc, _9n_TestBloomFilteredLucenePostings_0.tim, _9n_TestBloomFilteredLucenePostings_0.tip, _9p.cfe, _9p.cfs, _9p.si, _9t.cfe, _9t.cfs, _9t.si, _9w.cfe, _9w.cfs, _9w.si, _9x.cfe, _9x.cfs, _9x.si, _a0.cfe, _a0.cfs, _a0.si, _a1.cfe, _a1.cfs, _a1.si, _a2.cfe, _a2.cfs, _a2.si, _a3.cfe, _a3.cfs, _a3.si, _a4.cfe, _a4.cfs, _a4.si, _a5.cfe, _a5.cfs, _a5.si, _a6.cfe, _a6.cfs, _a6.si, _a7.cfe, _a7.cfs, _a7.si, _a8.cfe, _a8.cfs, _a8.si, _a9.cfe, _a9.cfs, _a9.si, _aa.cfe, _aa.cfs, _aa.si, _ab.cfe, _ab.cfs, _ab.si, _ac.cfe, _ac.cfs, _ac.si, _ad.cfe, _ad.cfs, _ad.si, _ae.cfe, _ae.cfs, _ae.si, _af.cfe, _af.cfs, _af.si, _ag.cfe, _ag.cfs, _ag.si, _ah.cfe, _ah.cfs, _ah.si, _ai.cfe, _ai.cfs, _ai.si, _aj.cfe, _aj.cfs, _aj.si, _ak.cfe, _ak.cfs, _ak.si, segments_2]}]> but was:<[{indexVersion=1494891787746,generation=2,filelist=[_9i.cfe, _9i.cfs, _9i.si, _9n.fdt, _9n.fdx, _9n.fnm, _9n.nvd, _9n.nvm, _9n.si, _9n_TestBloomFilteredLucenePostings_0.blm, _9n_TestBloomFilteredLucenePostings_0.doc, _9n_TestBloomFilteredLucenePostings_0.tim, _9n_TestBloomFilteredLucenePostings_0.tip, _9p.cfe, _9p.cfs, _9p.si, _9t.cfe, _9t.cfs, _9t.si, _9w.cfe, _9w.cfs, _9w.si, _9x.cfe, _9x.cfs, _9x.si, _a0.cfe, _a0.cfs, _a0.si, _a1.cfe, _a1.cfs, _a1.si, _a2.cfe, _a2.cfs, _a2.si, _a3.cfe, _a3.cfs, _a3.si, _a4.cfe, _a4.cfs, _a4.si, _a5.cfe, _a5.cfs, _a5.si, _a6.cfe, _a6.cfs, _a6.si, _a7.cfe, _a7.cfs, _a7.si, _a8.cfe, _a8.cfs, _a8.si, _a9.cfe, _a9.cfs, _a9.si, _aa.cfe, _aa.cfs, _aa.si, _ab.cfe, _ab.cfs, _ab.si, _ac.cfe, _ac.cfs, _ac.si, _ad.cfe, _ad.cfs, _ad.si, _ae.cfe, _ae.cfs, _ae.si, _af.cfe, _af.cfs, _af.si, _ag.cfe, _ag.cfs, _ag.si, _ah.cfe, _ah.cfs, _ah.si, _ai.cfe, _ai.cfs, _ai.si, _aj.cfe, _aj.cfs, _aj.si, _ak.cfe, _ak.cfs, _ak.si, segments_2]}, {indexVersion=1494891787746,generation=3,filelist=[_9i.cfe, _9i.cfs, _9i.si, _9p.cfe, _9p.cfs, _9p.si, _9t.cfe, _9t.cfs, _9t.si, _9w.cfe, _9w.cfs, _9w.si, _9x.cfe, _9x.cfs, _9x.si, _a0.cfe, _a0.cfs, _a0.si, _a1.cfe, _a1.cfs, _a1.si, _a3.cfe, _a3.cfs, _a3.si, _a4.cfe, _a4.cfs, _a4.si, _a5.cfe, _a5.cfs, _a5.si, _a6.cfe, _a6.cfs, _a6.si, _a7.cfe, _a7.cfs, _a7.si, _a8.cfe, _a8.cfs, _a8.si, _a9.cfe, _a9.cfs, _a9.si, _aa.cfe, _aa.cfs, _aa.si, _ab.cfe, _ab.cfs, _ab.si, _ac.cfe, _ac.cfs, _ac.si, _ad.cfe, _ad.cfs, _ad.si, _ag.cfe, _ag.cfs, _ag.si, _ah.cfe, _ah.cfs, _ah.si, _ai.cfe, _ai.cfs, _ai.si, _ak.cfe, _ak.cfs, _ak.si, _al.cfe, _al.cfs, _al.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1494891787746,generation=2,filelist=[_9i.cfe, _9i.cfs, _9i.si, _9n.fdt, _9n.fdx, _9n.fnm, _9n.nvd, _9n.nvm, _9n.si, _9n_TestBloomFilteredLucenePostings_0.blm, _9n_TestBloomFilteredLucenePostings_0.doc, _9n_TestBloomFilteredLucenePostings_0.tim, _9n_TestBloomFilteredLucenePostings_0.tip, _9p.cfe, _9p.cfs, _9p.si, _9t.cfe, _9t.cfs, _9t.si, _9w.cfe, _9w.cfs, _9w.si, _9x.cfe, _9x.cfs, _9x.si, _a0.cfe, _a0.cfs, _a0.si, _a1.cfe, _a1.cfs, _a1.si, _a2.cfe, _a2.cfs, _a2.si, _a3.cfe, _a3.cfs, _a3.si, _a4.cfe, _a4.cfs, _a4.si, _a5.cfe, _a5.cfs, _a5.si, _a6.cfe, _a6.cfs, _a6.si, _a7.cfe, _a7.cfs, _a7.si, _a8.cfe, _a8.cfs, _a8.si, _a9.cfe, _a9.cfs, _a9.si, _aa.cfe, _aa.cfs, _aa.si, _ab.cfe, _ab.cfs, _ab.si, _ac.cfe, _ac.cfs, _ac.si, _ad.cfe, _ad.cfs, _ad.si, _ae.cfe, _ae.cfs, _ae.si, _af.cfe, _af.cfs, _af.si, _ag.cfe, _ag.cfs, _ag.si, _ah.cfe, _ah.cfs, _ah.si, _ai.cfe, _ai.cfs, _ai.si, _aj.cfe, _aj.cfs, _aj.si, _ak.cfe, _ak.cfs, _ak.si, segments_2]}]> but was:<[{indexVersion=1494891787746,generation=2,filelist=[_9i.cfe, _9i.cfs, _9i.si, _9n.fdt, _9n.fdx, _9n.fnm, _9n.nvd, _9n.nvm, _9n.si, _9n_TestBloomFilteredLucenePostings_0.blm, _9n_TestBloomFilteredLucenePostings_0.doc, _9n_TestBloomFilteredLucenePostings_0.tim, _9n_TestBloomFilteredLucenePostings_0.tip, _9p.cfe, _9p.cfs, _9p.si, _9t.cfe, _9t.cfs, _9t.si, _9w.cfe, _9w.cfs, _9w.si, _9x.cfe, _9x.cfs, _9x.si, _a0.cfe, _a0.cfs, _a0.si, _a1.cfe, _a1.cfs, _a1.si, _a2.cfe, _a2.cfs, _a2.si, _a3.cfe, _a3.cfs, _a3.si, _a4.cfe, _a4.cfs, _a4.si, _a5.cfe, _a5.cfs, _a5.si, _a6.cfe, _a6.cfs, _a6.si, _a7.cfe, _a7.cfs, _a7.si, _a8.cfe, _a8.cfs, _a8.si, _a9.cfe, _a9.cfs, _a9.si, _aa.cfe, _aa.cfs, _aa.si, _ab.cfe, _ab.cfs, _ab.si, _ac.cfe, _ac.cfs, _ac.si, _ad.cfe, _ad.cfs, _ad.si, _ae.cfe, _ae.cfs, _ae.si, _af.cfe, _af.cfs, _af.si, _ag.cfe, _ag.cfs, _ag.si, _ah.cfe, _ah.cfs, _ah.si, _ai.cfe, _ai.cfs, _ai.si, _aj.cfe, _aj.cfs, _aj.si, _ak.cfe, _ak.cfs, _ak.si, segments_2]}, {indexVersion=1494891787746,generation=3,filelist=[_9i.cfe, _9i.cfs, _9i.si, _9p.cfe, _9p.cfs, _9p.si, _9t.cfe, _9t.cfs, _9t.si, _9w.cfe, _9w.cfs, _9w.si, _9x.cfe, _9x.cfs, _9x.si, _a0.cfe, _a0.cfs, _a0.si, _a1.cfe, _a1.cfs, _a1.si, _a3.cfe, _a3.cfs, _a3.si, _a4.cfe, _a4.cfs, _a4.si, _a5.cfe, _a5.cfs, _a5.si, _a6.cfe, _a6.cfs, _a6.si, _a7.cfe, _a7.cfs, _a7.si, _a8.cfe, _a8.cfs, _a8.si, _a9.cfe, _a9.cfs, _a9.si, _aa.cfe, _aa.cfs, _aa.si, _ab.cfe, _ab.cfs, _ab.si, _ac.cfe, _ac.cfs, _ac.si, _ad.cfe, _ad.cfs, _ad.si, _ag.cfe, _ag.cfs, _ag.si, _ah.cfe, _ah.cfs, _ah.si, _ai.cfe, _ai.cfs, _ai.si, _ak.cfe, _ak.cfs, _ak.si, _al.cfe, _al.cfs, _al.si, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([A06181A33B6CB8AD:85B69A934B24B6AE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1279)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11757 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/init-core-data-001
   [junit4]   2> 393  INFO  (SUITE-TestReplicationHandler-seed#[A06181A33B6CB8AD]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 404  INFO  (SUITE-TestReplicationHandler-seed#[A06181A33B6CB8AD]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 407  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestIndexFetchWithMasterUrl
   [junit4]   2> 408  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/collection1
   [junit4]   2> 865  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 936  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3478a444{/solr,null,AVAILABLE}
   [junit4]   2> 955  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@354f9538{HTTP/1.1,[http/1.1]}{127.0.0.1:58880}
   [junit4]   2> 956  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.Server Started @4015ms
   [junit4]   2> 956  INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=58880}
   [junit4]   2> 1000 ERROR (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1000 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1001 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1001 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1067 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-15T23:37:47.592Z
   [junit4]   2> 1098 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1106 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/solr.xml
   [junit4]   2> 2253 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/.
   [junit4]   2> 2263 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2428 WARN  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 2438 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2608 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2766 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2767 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 2857 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2863 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 2922 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=50.1328125, floorSegmentMB=0.611328125, forceMergeDeletesPctAllowed=21.56046739139764, segmentsPerTier=13.0, maxCFSSegmentSizeMB=2.154296875, noCFSRatio=0.648647152513234
   [junit4]   2> 3826 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3827 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3852 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2c7cdd53
   [junit4]   2> 3895 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@38d628ae[collection1] main]
   [junit4]   2> 3933 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-001/collection1/conf
   [junit4]   2> 3994 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 3995 INFO  (coreLoadExecutor-6-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4199 INFO  (searcherExecutor-7-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@38d628ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4444 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1
   [junit4]   2> 4452 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 4557 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@637c8868{/solr,null,AVAILABLE}
   [junit4]   2> 4558 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@374b6047{HTTP/1.1,[http/1.1]}{127.0.0.1:37587}
   [junit4]   2> 4558 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.Server Started @7616ms
   [junit4]   2> 4558 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=37587}
   [junit4]   2> 4558 ERROR (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4576 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 4577 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4577 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4577 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-15T23:37:51.165Z
   [junit4]   2> 4578 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/solr.xml
   [junit4]   2> 4683 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/.
   [junit4]   2> 4684 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4774 WARN  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 4775 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 4807 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4855 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 4856 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 4857 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 4857 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 4868 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=50.1328125, floorSegmentMB=0.611328125, forceMergeDeletesPctAllowed=21.56046739139764, segmentsPerTier=13.0, maxCFSSegmentSizeMB=2.154296875, noCFSRatio=0.648647152513234
   [junit4]   2> 4979 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4986 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4988 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@464e0aa0
   [junit4]   2> 4990 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3581769b[collection1] main]
   [junit4]   2> 4992 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1/conf
   [junit4]   2> 5031 INFO  (coreLoadExecutor-16-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 5032 INFO  (searcherExecutor-17-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3581769b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5288 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@374b6047{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 5289 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2111753076
   [junit4]   2> 5289 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 5290 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 5290 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 5291 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 5295 INFO  (coreCloseExecutor-22-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7a73e46a
   [junit4]   2> 5327 INFO  (coreCloseExecutor-22-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2054415466
   [junit4]   2> 5338 INFO  (indexFetcher-21-thread-1) [    x:collection1] o.a.h.i.e.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}-><a href="http://127.0.0.1:58880:">http://127.0.0.1:58880: The target server failed to respond
   [junit4]   2> 5339 INFO  (indexFetcher-21-thread-1) [    x:collection1] o.a.h.i.e.RetryExec Retrying request to {}->http://127.0.0.1:58880
   [junit4]   2> 5340 WARN  (indexFetcher-21-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master at: http://127.0.0.1:58880/solr/collection1 is not available. Index fetch failed by exception: java.lang.IllegalStateException: Connection pool shut down
   [junit4]   2> 5342 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@637c8868{/solr,null,UNAVAILABLE}
   [junit4]   2> 5345 WARN  (indexFetcher-21-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Could not log failed replication details
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: Already closed
   [junit4]   2> at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:336)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.loadReplicationProperties(ReplicationHandler.java:1127)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.logReplicationTimeAndConfFiles(IndexFetcher.java:746)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.cleanup(IndexFetcher.java:654)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:636)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:326)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:415)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1179)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 5349 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 5360 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4d0823f1{/solr,null,AVAILABLE}
   [junit4]   2> 5360 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7df87318{HTTP/1.1,[http/1.1]}{127.0.0.1:46698}
   [junit4]   2> 5361 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.Server Started @8419ms
   [junit4]   2> 5361 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=46698}
   [junit4]   2> 5361 ERROR (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 5361 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 5361 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 5362 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 5362 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-15T23:37:51.950Z
   [junit4]   2> 5362 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/solr.xml
   [junit4]   2> 5482 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/.
   [junit4]   2> 5482 INFO  (TEST-TestReplicationHandler.doTestIndexFetchWithMasterUrl-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 5505 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=32
   [junit4]   2> 5571 WARN  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 5572 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 5642 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 5653 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 5657 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 5658 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 5658 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 5663 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=50.1328125, floorSegmentMB=0.611328125, forceMergeDeletesPctAllowed=21.56046739139764, segmentsPerTier=13.0, maxCFSSegmentSizeMB=2.154296875, noCFSRatio=0.648647152513234
   [junit4]   2> 5847 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5847 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5850 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5e4da47a
   [junit4]   2> 5853 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2acbb03f[collection1] main]
   [junit4]   2> 5854 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001/solr-instance-002/collection1/conf
   [junit4]   2> 5856 INFO  (coreLoadExecutor-29-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5857 INFO  (searcherExecutor-30-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2acbb03f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6127 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 6129 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 98
   [junit4]   2> 6160 INFO  (qtp1244307652-56) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 6162 INFO  (qtp1244307652-56) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 3
   [junit4]   2> 6169 INFO  (qtp1244307652-59) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 6169 INFO  (qtp1244307652-59) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1ea3fd45 commitCommandVersion:0
   [junit4]   2> 6175 INFO  (qtp1244307652-59) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@28602599[collection1] main]
   [junit4]   2> 6177 INFO  (qtp1244307652-59) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 6178 INFO  (searcherExecutor-30-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@28602599[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6180 INFO  (qtp1244307652-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 12
   [junit4]   2> 6383 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 199
   [junit4]   2> 6389 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 1
   [junit4]   2> 6393 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 6397 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 1
   [junit4]   2> 6400 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 6409 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 5
   [junit4]   2> 6414 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 1
   [junit4]   2> 6422 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 1
   [junit4]   2> 6427 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 1
   [junit4]   2> 6434 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 1
   [junit4]   2> 6439 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 1
   [junit4]   2> 6442 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 1
   [junit4]   2> 6447 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 1
   [junit4]   2> 6451 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 1
   [junit4]   2> 6487 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 6502 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 6
   [junit4]   2> 6506 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 1
   [junit4]   2> 6516 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 6527 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 6530 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 6531 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 6533 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 6535 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 6537 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 6539 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 6543 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 6545 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 6547 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 6556 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 6558 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 6560 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 6562 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 6565 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 1
   [junit4]   2> 6569 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 6573 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 6588 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 6595 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 6597 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 6599 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 6601 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 6602 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 6604 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 6606 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 6609 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 6611 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 6612 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 6614 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 6616 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 6618 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 6623 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 2
   [junit4]   2> 6625 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 6628 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 6630 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 6633 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 6635 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 6637 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 6639 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 6641 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 6643 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 6645 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 6647 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 6649 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 6651 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 6654 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 6656 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 6658 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 6660 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 6662 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 6677 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 6679 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 6681 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 6683 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 6685 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 6690 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 6691 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 6693 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 6712 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 6716 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 6718 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 6721 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 6723 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 6726 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 6732 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 6734 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 6735 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 6737 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 6739 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 6741 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 6743 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 6745 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 6747 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 6751 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 6753 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 6755 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 6760 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 6762 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 6764 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 6772 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 6777 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 6779 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 6780 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 6782 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 6792 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 6794 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 6796 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 6797 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 6799 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 6801 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 6802 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 6804 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 6808 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 6810 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 6812 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 6820 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 6822 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 6824 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 6825 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 6827 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 6829 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 6830 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 6832 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 6834 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 6836 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 6841 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 6844 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 6847 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 6848 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 6924 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 74
   [junit4]   2> 6930 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 1
   [junit4]   2> 6932 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 6940 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 6942 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 6950 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 1
   [junit4]   2> 6953 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 6973 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 1
   [junit4]   2> 6978 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 1
   [junit4]   2> 7008 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 7010 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 7011 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 7018 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 3
   [junit4]   2> 7020 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 7024 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 7027 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 7030 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 0
   [junit4]   2> 7033 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 7044 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 7046 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 7048 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 7060 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 7062 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 7064 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
   [junit4]   2> 7066 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 0
   [junit4]   2> 7067 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
   [junit4]   2> 7069 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 0
   [junit4]   2> 7071 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
   [junit4]   2> 7073 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
   [junit4]   2> 7074 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
   [junit4]   2> 7076 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
   [junit4]   2> 7077 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
   [junit4]   2> 7079 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 0
   [junit4]   2> 7081 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
   [junit4]   2> 7082 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
   [junit4]   2> 7084 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
   [junit4]   2> 7088 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
   [junit4]   2> 7093 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
   [junit4]   2> 7100 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
   [junit4]   2> 7109 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 0
   [junit4]   2> 7111 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[167]} 0 0
   [junit4]   2> 7112 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[168]} 0 0
   [junit4]   2> 7114 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[169]} 0 0
   [junit4]   2> 7115 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[170]} 0 0
   [junit4]   2> 7117 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[171]} 0 0
   [junit4]   2> 7119 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[172]} 0 0
   [junit4]   2> 7120 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[173]} 0 0
   [junit4]   2> 7122 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[174]} 0 0
   [junit4]   2> 7123 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[175]} 0 0
   [junit4]   2> 7125 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[176]} 0 0
   [junit4]   2> 7126 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[177]} 0 0
   [junit4]   2> 7128 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[178]} 0 0
   [junit4]   2> 7129 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[179]} 0 0
   [junit4]   2> 7131 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[180]} 0 0
   [junit4]   2> 7132 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[181]} 0 0
   [junit4]   2> 7134 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[182]} 0 0
   [junit4]   2> 7136 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[183]} 0 0
   [junit4]   2> 7137 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[184]} 0 0
   [junit4]   2> 7139 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[185]} 0 0
   [junit4]   2> 7140 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[186]} 0 0
   [junit4]   2> 7142 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[187]} 0 0
   [junit4]   2> 7144 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[188]} 0 0
   [junit4]   2> 7145 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[189]} 0 0
   [junit4]   2> 7147 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[190]} 0 0
   [junit4]   2> 7149 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[191]} 0 0
   [junit4]   2> 7150 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[192]} 0 0
   [junit4]   2> 7152 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[193]} 0 0
   [junit4]   2> 7153 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[194]} 0 0
   [junit4]   2> 7155 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[195]} 0 0
   [junit4]   2> 7157 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[196]} 0 0
   [junit4]   2> 7158 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[197]} 0 0
   [junit4]   2> 7160 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[198]} 0 0
   [junit4]   2> 7169 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[199]} 0 0
   [junit4]   2> 7170 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[200]} 0 0
   [junit4]   2> 7172 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[201]} 0 0
   [junit4]   2> 7174 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[202]} 0 0
   [junit4]   2> 7188 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[203]} 0 0
   [junit4]   2> 7190 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[204]} 0 0
   [junit4]   2> 7192 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[205]} 0 0
   [junit4]   2> 7194 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[206]} 0 0
   [junit4]   2> 7195 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[207]} 0 0
   [junit4]   2> 7197 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[208]} 0 0
   [junit4]   2> 7199 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[209]} 0 0
   [junit4]   2> 7201 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[210]} 0 0
   [junit4]   2> 7208 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[211]} 0 0
   [junit4]   2> 7210 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[212]} 0 0
   [junit4]   2> 7216 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[213]} 0 0
   [junit4]   2> 7230 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[214]} 0 0
   [junit4]   2> 7232 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[215]} 0 0
   [junit4]   2> 7234 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[216]} 0 0
   [junit4]   2> 7236 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[217]} 0 0
   [junit4]   2> 7239 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[218]} 0 0
   [junit4]   2> 7242 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[219]} 0 0
   [junit4]   2> 7248 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[220]} 0 0
   [junit4]   2> 7261 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[221]} 0 0
   [junit4]   2> 7273 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[222]} 0 0
   [junit4]   2> 7278 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[223]} 0 0
   [junit4]   2> 7283 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[224]} 0 0
   [junit4]   2> 7284 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[225]} 0 0
   [junit4]   2> 7296 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[226]} 0 0
   [junit4]   2> 7300 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[227]} 0 0
   [junit4]   2> 7302 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[228]} 0 0
   [junit4]   2> 7312 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[229]} 0 0
   [junit4]   2> 7320 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[230]} 0 0
   [junit4]   2> 7322 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[231]} 0 0
   [junit4]   2> 7324 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[232]} 0 0
   [junit4]   2> 7330 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[233]} 0 0
   [junit4]   2> 7338 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[234]} 0 0
   [junit4]   2> 7340 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[235]} 0 0
   [junit4]   2> 7344 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[236]} 0 0
   [junit4]   2> 7347 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[237]} 0 0
   [junit4]   2> 7350 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[238]} 0 0
   [junit4]   2> 7361 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[239]} 0 0
   [junit4]   2> 7364 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[240]} 0 0
   [junit4]   2> 7366 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[241]} 0 0
   [junit4]   2> 7367 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[242]} 0 0
   [junit4]   2> 7369 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[243]} 0 0
   [junit4]   2> 7372 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[244]} 0 0
   [junit4]   2> 7374 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[245]} 0 0
   [junit4]   2> 7376 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[246]} 0 0
   [junit4]   2> 7393 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[247]} 0 0
   [junit4]   2> 7396 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[248]} 0 0
   [junit4]   2> 7398 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[249]} 0 0
   [junit4]   2> 7400 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[250]} 0 0
   [junit4]   2> 7407 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[251]} 0 0
   [junit4]   2> 7409 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[252]} 0 0
   [junit4]   2> 7412 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[253]} 0 0
   [junit4]   2> 7423 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[254]} 0 0
   [junit4]   2> 7443 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[255]} 0 18
   [junit4]   2> 7446 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[256]} 0 0
   [junit4]   2> 7448 INFO  (qtp1637397128-21) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[257]} 0 0
   [junit4]   2> 7450 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[258]} 0 0
   [junit4]   2> 7452 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[259]} 0 0
   [junit4]   2> 7455 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[260]} 0 0
   [junit4]   2> 7457 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[261]} 0 0
   [junit4]   2> 7459 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[262]} 0 0
   [junit4]   2> 7460 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[263]} 0 0
   [junit4]   2> 7462 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[264]} 0 0
   [junit4]   2> 7463 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[265]} 0 0
   [junit4]   2> 7466 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[266]} 0 0
   [junit4]   2> 7467 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[267]} 0 0
   [junit4]   2> 7469 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[268]} 0 0
   [junit4]   2> 7470 INFO  (qtp1637397128-23) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[269]} 0 0
   [junit4]   2> 7472 INFO  (qtp1637397128-16) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[270]} 0 0
   [junit4]   2> 7474 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[271]} 0 0
   [junit4]   2> 7483 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[272]} 0 0
   [junit4]   2> 7485 INFO  (qtp1637397128-20) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[273]} 0 0
   [junit4]   2> 7486 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[274]} 0 0
   [junit4]   2> 7488 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[275]} 0 0
   [junit4]   2> 7489 INFO  (qtp1637397128-17) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[276]} 0 0
   [junit4]   2> 7491 INFO  (qtp1637397128-22) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[277

[...truncated too long message...]

Fetcher Skipping download for _3.si because it already exists
   [junit4]   2> 335012 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _3.cfs because it already exists
   [junit4]   2> 335012 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _3.cfe because it already exists
   [junit4]   2> 335013 INFO  (qtp1030944918-1623) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=5&qt=/replication&file=_4.cfe&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 335015 INFO  (qtp1030944918-1624) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=5&qt=/replication&file=_4.cfs&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 335016 INFO  (qtp1030944918-1625) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=5&qt=/replication&file=_4.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 335018 INFO  (qtp1030944918-1619) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=5&qt=/replication&file=segments_5&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 335019 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=2488) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@2490340e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b73d580)
   [junit4]   2> 335023 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=7, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 335026 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 335029 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5e7547a6[collection1] main]
   [junit4]   2> 335030 INFO  (searcherExecutor-881-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5e7547a6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(7.0.0):c2) Uninverting(_3(7.0.0):c1) Uninverting(_4(7.0.0):c1)))}
   [junit4]   2> 335030 INFO  (qtp1071574142-1639) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:39336/solr/collection1/replication&command=fetchindex} status=0 QTime=131
   [junit4]   2> 335049 INFO  (qtp1030944918-1622) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=name:(empty1+empty2+empty3+empty4)&sort=id+desc&wt=javabin&version=2} hits=4 status=0 QTime=18
   [junit4]   2> 335051 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 4 docs
   [junit4]   2> 335053 INFO  (qtp1071574142-1635) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=name:(empty1+empty2+empty3+empty4)&sort=id+desc&wt=javabin&version=2} hits=4 status=0 QTime=1
   [junit4]   2> 335054 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 4 docs
   [junit4]   2> 335055 INFO  (qtp1030944918-1618) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 335056 INFO  (qtp1030944918-1623) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 335058 INFO  (qtp1030944918-1621) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 335058 INFO  (qtp1071574142-1640) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=1
   [junit4]   2> 335058 INFO  (qtp1030944918-1624) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 335059 INFO  (qtp1071574142-1642) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 335059 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEmptyCommits
   [junit4]   2> 335060 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4036b0b8{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 335060 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=994993626
   [junit4]   2> 335060 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 335060 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 335060 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 335061 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 335061 INFO  (coreCloseExecutor-890-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4be36a45
   [junit4]   2> 335063 INFO  (coreCloseExecutor-890-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1273195077
   [junit4]   2> 335063 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5bb5cac{/solr,null,UNAVAILABLE}
   [junit4]   2> 335066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e38ee31{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 335066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=239555375
   [junit4]   2> 335066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 335066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 335067 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 335067 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 335067 INFO  (coreCloseExecutor-892-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3ba3a751
   [junit4]   2> 335069 INFO  (coreCloseExecutor-892-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1000580945
   [junit4]   2> 335070 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[A06181A33B6CB8AD]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5bbafd44{/solr,null,UNAVAILABLE}
   [junit4]   2> 335072 INFO  (SUITE-TestReplicationHandler-seed#[A06181A33B6CB8AD]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_A06181A33B6CB8AD-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), newname=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{}, maxPointsInLeafNode=185, maxMBSortInHeap=5.004376588095391, sim=RandomSimilarity(queryNorm=true): {}, locale=es-PR, timezone=Africa/Timbuktu
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=17562024,total=322961408
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudInspectUtil, TestReplicationHandler]
   [junit4] Completed [73/708 (1!)] on J0 in 334.75s, 16 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 45424 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1302 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1302/

5 tests failed.
FAILED:  org.apache.lucene.search.TestDoubleRangeFieldQueries.testRandomTiny

Error Message:
wrong hit (first of possibly more):  FAIL (iter 31): id=1 should not match but did  queryRange=Box(-Infinity TO Infinity)  box=Box(-5.862528225813263E307 TO 7.363410523584262E307)  queryType=CONTAINS  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 31): id=1 should not match but did
 queryRange=Box(-Infinity TO Infinity)
 box=Box(-5.862528225813263E307 TO 7.363410523584262E307)
 queryType=CONTAINS
 deleted?=false
        at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.TestFloatRangeFieldQueries.testRandomTiny

Error Message:
wrong hit (first of possibly more):  FAIL (iter 31): id=1 should not match but did  queryRange=Box(-Infinity TO Infinity)  box=Box(-5.450555E37 TO 9.067386E37)  queryType=CONTAINS  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 31): id=1 should not match but did
 queryRange=Box(-Infinity TO Infinity)
 box=Box(-5.450555E37 TO 9.067386E37)
 queryType=CONTAINS
 deleted?=false
        at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.TestIntRangeFieldQueries.testRandomTiny

Error Message:
wrong hit (first of possibly more):  FAIL (iter 31): id=1 should not match but did  queryRange=Box(-2147483648 TO 2147483647)  box=Box(-1845312013 TO -1215188953)  queryType=CONTAINS  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 31): id=1 should not match but did
 queryRange=Box(-2147483648 TO 2147483647)
 box=Box(-1845312013 TO -1215188953)
 queryType=CONTAINS
 deleted?=false
        at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.TestLongRangeFieldQueries.testRandomTiny

Error Message:
wrong hit (first of possibly more):  FAIL (iter 31): id=1 should not match but did  queryRange=Box(-9223372036854775808 TO 9223372036854775807)  box=Box(-1803576629531660249 TO 7946735353577260531)  queryType=CONTAINS  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 31): id=1 should not match but did
 queryRange=Box(-9223372036854775808 TO 9223372036854775807)
 box=Box(-1803576629531660249 TO 7946735353577260531)
 queryType=CONTAINS
 deleted?=false
        at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
        at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.test

Error Message:
There are still nodes recoverying - waited for 30 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
        at __randomizedtesting.SeedInfo.seed([4BCC2C9A43BDECB1:C3981340ED418149]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:187)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:867)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1423)
        at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 798 lines...]
   [junit4] Suite: org.apache.lucene.search.TestLongRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLongRangeFieldQueries -Dtests.method=testRandomTiny -Dtests.seed=7FAAAB755DD26282 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-CU -Dtests.timezone=Brazil/DeNoronha -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.02s J2 | TestLongRangeFieldQueries.testRandomTiny <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 31): id=1 should not match but did
   [junit4]    >  queryRange=Box(-9223372036854775808 TO 9223372036854775807)
   [junit4]    >  box=Box(-1803576629531660249 TO 7946735353577260531)
   [junit4]    >  queryType=CONTAINS
   [junit4]    >  deleted?=false
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.search.TestLongRangeFieldQueries_7FAAAB755DD26282-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=es-CU, timezone=Brazil/DeNoronha
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=198187128,total=399507456
   [junit4]   2> NOTE: All tests run in this JVM: [TestSortedSetSortField, TestIntArrayDocIdSet, TestIOUtils, TestSpanBoostQuery, TestUsageTrackingFilterCachingPolicy, TestFilterDirectoryReader, TestAllFilesHaveChecksumFooter, TestTopDocsCollector, TestNIOFSDirectory, TestSimilarity2, TestBoolean2ScorerSupplier, TestPayloads, TestStopFilter, TestLogMergePolicy, TestIndexSearcher, TestMixedCodecs, Test2BSortedDocValuesOrds, TestLucene50StoredFieldsFormatHighCompression, TestDocIDMerger, TestRegexpQuery, TestMathUtil, TestIndexCommit, TestMultiset, TestSetOnce, TestDisjunctionMaxQuery, Test2BPagedBytes, TestHighCompressionMode, TestBlockPostingsFormat, TestConsistentFieldNumbers, TestSameTokenSamePosition, TestLucene70NormsFormat, TestMultiLevelSkipList, TestSearchForDuplicates, TestMinShouldMatch2, TestNRTReaderWithThreads, TestSpanTermQuery, TestSloppyPhraseQuery2, TestStressAdvance, TestSleepingLockWrapper, TestSegmentReader, TestSimpleExplanationsOfNonMatches, TestLazyProxSkipping, TestSimpleFSDirectory, TestEarlyTerminatingSortingCollector, TestLSBRadixSorter, TestSpanSearchEquivalence, TestOneMergeWrappingMergePolicy, TestIndexWriterDeleteByQuery, TestBytesRefHash, TestDirectMonotonic, TestFilterWeight, TestCharsRef, TestIndexWriterMerging, TestExitableDirectoryReader, TestIndexWriterThreadsToSegments, Test2BPositions, TestHugeRamFile, TestFixedBitSet, TestComplexExplanationsOfNonMatches, TestPayloadsOnVectors, MultiCollectorTest, TestRollingBuffer, TestIndexWriterLockRelease, TestFieldsReader, TestFastCompressionMode, TestSpanNearQuery, TestDateSort, TestSpans, TestIndexWriterCommit, TestNoMergePolicy, TestFixedLengthBytesRefArray, TestMultiMMap, TestSortRescorer, TestLongRangeFieldQueries]
   [junit4] Completed [180/451 (1!)] on J2 in 54.36s, 5 tests, 1 failure <<< FAILURES!

[...truncated 182 lines...]
   [junit4] Suite: org.apache.lucene.search.TestDoubleRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDoubleRangeFieldQueries -Dtests.method=testRandomTiny -Dtests.seed=7FAAAB755DD26282 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=bg-BG -Dtests.timezone=US/Eastern -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.06s J0 | TestDoubleRangeFieldQueries.testRandomTiny <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 31): id=1 should not match but did
   [junit4]    >  queryRange=Box(-Infinity TO Infinity)
   [junit4]    >  box=Box(-5.862528225813263E307 TO 7.363410523584262E307)
   [junit4]    >  queryType=CONTAINS
   [junit4]    >  deleted?=false
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0/temp/lucene.search.TestDoubleRangeFieldQueries_7FAAAB755DD26282-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{doubleRangeField=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=269, maxMBSortInHeap=5.775973196836048, sim=RandomSimilarity(queryNorm=true): {}, locale=bg-BG, timezone=US/Eastern
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=246094992,total=428867584
   [junit4]   2> NOTE: All tests run in this JVM: [TestNot, TestBytesStore, TestCrashCausesCorruptIndex, TestBooleanRewrites, TestBooleanMinShouldMatch, TestTerm, TestLucene50TermVectorsFormat, TestIndexedDISI, TestLucene70SegmentInfoFormat, TestMultiThreadTermVectors, TestFlushByRamOrCountsPolicy, TestManyFields, TestNeverDelete, TestGeoUtils, TestDocInverterPerFieldErrorInfo, TestAllFilesCheckIndexHeader, TestSearchAfter, TestPostingsOffsets, TestStressDeletes, TestDocValuesIndexing, TestDirectoryReaderReopen, TestUnicodeUtil, TestBoolean2, TestLiveFieldValues, TestSynonymQuery, TestCachingCollector, TestSortedNumericSortField, TestEarlyTermination, TestTermVectorsWriter, TestClassicSimilarity, TestUpgradeIndexMergePolicy, TestMaxTermFrequency, TestIsCurrent, TestReaderWrapperDVTypeCheck, TestSpanMultiTermQueryWrapper, TestThreadedForceMerge, TestDateTools, TestIndexingSequenceNumbers, TestIndexWriterOnVMError, Test4GBStoredFields, TestTerms, TestNoDeletionPolicy, TestSimpleExplanations, TestTotalHitCountCollector, TestNeedsScores, TestLongBitSet, TestOfflineSorter, TestDoubleRangeFieldQueries]
   [junit4] Completed [230/451 (2!)] on J0 in 83.39s, 5 tests, 1 failure <<< FAILURES!

[...truncated 596 lines...]
   [junit4] Suite: org.apache.lucene.search.TestFloatRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFloatRangeFieldQueries -Dtests.method=testRandomTiny -Dtests.seed=7FAAAB755DD26282 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=en -Dtests.timezone=Arctic/Longyearbyen -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.01s J0 | TestFloatRangeFieldQueries.testRandomTiny <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 31): id=1 should not match but did
   [junit4]    >  queryRange=Box(-Infinity TO Infinity)
   [junit4]    >  box=Box(-5.450555E37 TO 9.067386E37)
   [junit4]    >  queryType=CONTAINS
   [junit4]    >  deleted?=false
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0/temp/lucene.search.TestFloatRangeFieldQueries_7FAAAB755DD26282-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=FST50}, docValues:{floatRangeField=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=99, maxMBSortInHeap=7.97891397353937, sim=RandomSimilarity(queryNorm=true): {}, locale=en, timezone=Arctic/Longyearbyen
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=255445904,total=522715136
   [junit4]   2> NOTE: All tests run in this JVM: [TestNot, TestBytesStore, TestCrashCausesCorruptIndex, TestBooleanRewrites, TestBooleanMinShouldMatch, TestTerm, TestLucene50TermVectorsFormat, TestIndexedDISI, TestLucene70SegmentInfoFormat, TestMultiThreadTermVectors, TestFlushByRamOrCountsPolicy, TestManyFields, TestNeverDelete, TestGeoUtils, TestDocInverterPerFieldErrorInfo, TestAllFilesCheckIndexHeader, TestSearchAfter, TestPostingsOffsets, TestStressDeletes, TestDocValuesIndexing, TestDirectoryReaderReopen, TestUnicodeUtil, TestBoolean2, TestLiveFieldValues, TestSynonymQuery, TestCachingCollector, TestSortedNumericSortField, TestEarlyTermination, TestTermVectorsWriter, TestClassicSimilarity, TestUpgradeIndexMergePolicy, TestMaxTermFrequency, TestIsCurrent, TestReaderWrapperDVTypeCheck, TestSpanMultiTermQueryWrapper, TestThreadedForceMerge, TestDateTools, TestIndexingSequenceNumbers, TestIndexWriterOnVMError, Test4GBStoredFields, TestTerms, TestNoDeletionPolicy, TestSimpleExplanations, TestTotalHitCountCollector, TestNeedsScores, TestLongBitSet, TestOfflineSorter, TestDoubleRangeFieldQueries, TestPerSegmentDeletes, TestForceMergeForever, TestMultiTermQueryRewrites, TestUTF32ToUTF8, TestSimpleAttributeImpl, TestBytesRef, TestTopFieldCollector, TestSloppyPhraseQuery, TestDeterminizeLexicon, TestWordlistLoader, TestAutomatonQuery, TestReqExclBulkScorer, TestIndexWriterConfig, TestRoaringDocIdSet, TestSmallFloat, TestIndexableField, TestSort, TestSimpleSearchEquivalence, TestFrequencyTrackingRingBuffer, Test2BPoints, TestNRTCachingDirectory, TestRegexpRandom, TestNamedSPILoader, TestStressIndexing, TestGrowableByteArrayDataOutput, TestDocIdSetBuilder, TestCodecHoldsOpenFiles, TestOmitNorms, TestComplexExplanations, TestFastDecompressionMode, TestParallelCompositeReader, TestIndexWriterNRTIsCurrent, TestFieldCacheRewriteMethod, TestPagedBytes, TestSegmentTermDocs, TestPerFieldPostingsFormat2, TestMultiDocValues, TestSimilarityBase, TestTermsEnum2, TestFlex, TestRAMDirectory, TestCodecs, TestCharTermAttributeImpl, TestOmitPositions, TestOmitTf, TestNearSpansOrdered, TestTermRangeQuery, TestDocumentsWriterDeleteQueue, TestDocumentWriter, TestScorerPerf, TestDoc, TestFieldMaskingSpanQuery, TestTermVectorsReader, TestIntBlockPool, TestSegmentTermEnum, TestUniqueTermCount, TestBooleanQuery, TestAllFilesHaveCodecHeader, TestParallelReaderEmptyIndex, TestDocument, TestTermScorer, TestSumDocFreq, TestWildcardRandom, TestCheckIndex, TestBooleanQueryVisitSubscorers, TestCompiledAutomaton, TestMatchAllDocsQuery, TestBinaryTerms, TestSubScorerFreqs, TestRecyclingIntBlockAllocator, TestConjunctions, TestPositionIncrement, TestField, TestSimilarity, TestVirtualMethod, TestSimilarityProvider, TestFilterLeafReader, TestSearch, TestCachingTokenFilter, TestFilterIterator, Test2BPostings, TestExceedMaxTermLength, TestFieldReuse, TestIndexReaderClose, TestIndexWriterExceptions2, TestIndexWriterMaxDocs, TestMaxPosition, TestMergePolicyWrapper, TestMergeRateLimiter, TestMultiTermsEnum, TestNRTReaderCleanup, TestOrdinalMap, TestReadOnlyIndex, TestSortedSetDocValues, TestTermVectors, TestTragicIndexWriterDeadlock, TestTryDelete, TermInSetQueryTest, TestApproximationSearchEquivalence, TestConjunctionDISI, TestControlledRealTimeReopenThread, TestFloatRangeFieldQueries]
   [junit4] Completed [405/451 (3!)] on J0 in 67.95s, 5 tests, 1 failure <<< FAILURES!

[...truncated 15 lines...]
   [junit4] Suite: org.apache.lucene.search.TestIntRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIntRangeFieldQueries -Dtests.method=testRandomTiny -Dtests.seed=7FAAAB755DD26282 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=fr-LU -Dtests.timezone=America/Creston -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.01s J2 | TestIntRangeFieldQueries.testRandomTiny <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 31): id=1 should not match but did
   [junit4]    >  queryRange=Box(-2147483648 TO 2147483647)
   [junit4]    >  box=Box(-1845312013 TO -1215188953)
   [junit4]    >  queryType=CONTAINS
   [junit4]    >  deleted?=false
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7FAAAB755DD26282:36ED753303F35A2E]:0)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomTiny(BaseRangeFieldQueryTestCase.java:64)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.search.TestIntRangeFieldQueries_7FAAAB755DD26282-001
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=8, maxDocsPerChunk=90, blockSize=5), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=8, blockSize=5)), sim=RandomSimilarity(queryNorm=true): {}, locale=fr-LU, timezone=America/Creston
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=303091560,total=480247808
   [junit4]   2> NOTE: All tests run in this JVM: [TestSortedSetSortField, TestIntArrayDocIdSet, TestIOUtils, TestSpanBoostQuery, TestUsageTrackingFilterCachingPolicy, TestFilterDirectoryReader, TestAllFilesHaveChecksumFooter, TestTopDocsCollector, TestNIOFSDirectory, TestSimilarity2, TestBoolean2ScorerSupplier, TestPayloads, TestStopFilter, TestLogMergePolicy, TestIndexSearcher, TestMixedCodecs, Test2BSortedDocValuesOrds, TestLucene50StoredFieldsFormatHighCompression, TestDocIDMerger, TestRegexpQuery, TestMathUtil, TestIndexCommit, TestMultiset, TestSetOnce, TestDisjunctionMaxQuery, Test2BPagedBytes, TestHighCompressionMode, TestBlockPostingsFormat, TestConsistentFieldNumbers, TestSameTokenSamePosition, TestLucene70NormsFormat, TestMultiLevelSkipList, TestSearchForDuplicates, TestMinShouldMatch2, TestNRTReaderWithThreads, TestSpanTermQuery, TestSloppyPhraseQuery2, TestStressAdvance, TestSleepingLockWrapper, TestSegmentReader, TestSimpleExplanationsOfNonMatches, TestLazyProxSkipping, TestSimpleFSDirectory, TestEarlyTerminatingSortingCollector, TestLSBRadixSorter, TestSpanSearchEquivalence, TestOneMergeWrappingMergePolicy, TestIndexWriterDeleteByQuery, TestBytesRefHash, TestDirectMonotonic, TestFilterWeight, TestCharsRef, TestIndexWriterMerging, TestExitableDirectoryReader, TestIndexWriterThreadsToSegments, Test2BPositions, TestHugeRamFile, TestFixedBitSet, TestComplexExplanationsOfNonMatches, TestPayloadsOnVectors, MultiCollectorTest, TestRollingBuffer, TestIndexWriterLockRelease, TestFieldsReader, TestFastCompressionMode, TestSpanNearQuery, TestDateSort, TestSpans, TestIndexWriterCommit, TestNoMergePolicy, TestFixedLengthBytesRefArray, TestMultiMMap, TestSortRescorer, TestLongRangeFieldQueries, TestAutomaton, TestTimSorterWorstCase, TestIndexWriter, TestIndexWriterReader, TestPackedInts, TestIndexWriterWithThreads, TestGraphTokenizers, TestShardSearching, TestIndexWriterExceptions, TestDeletionPolicy, TestBasics, TestIndexWriterMergePolicy, TestNRTThreads, TestLongPostings, TestMinimize, TestSnapshotDeletionPolicy, TestIndexWriterUnicode, TestIndexWriterForceMerge, TestPerFieldPostingsFormat, TestBufferedIndexInput, TestTermsEnum, TestGeoEncodingUtils, TestPolygon, TestPolygon2D, TestAllFilesDetectTruncation, TestBinaryDocValuesUpdates, TestDemoParallelLeafReader, TestDocsWithFieldSet, TestDuelingCodecsAtNight, TestDocValuesQueries, TestDocValuesRewriteMethod, TestDoubleValuesSource, TestFieldValueQuery, TestIndexOrDocValuesQuery, TestIntRangeFieldQueries]
   [junit4] Completed [410/451 (4!)] on J2 in 84.69s, 5 tests, 1 failure <<< FAILURES!

[...truncated 11695 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsSyncSliceTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/init-core-data-001
   [junit4]   2> 3263782 WARN  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1254 numCloses=1254
   [junit4]   2> 3263782 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 3263784 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3263785 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_aan/n
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3263831 WARN  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3263846 WARN  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3263850 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3263869 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_58796_hdfs____.4p0k79/webapp
   [junit4]   2> 3264228 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:58796
   [junit4]   2> 3264306 WARN  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3264307 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3264322 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34083_datanode____hlnx8x/webapp
   [junit4]   2> 3264711 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34083
   [junit4]   2> 3264760 WARN  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3264761 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3264781 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46818_datanode____.rayrjw/webapp
   [junit4]   2> 3264850 INFO  (IPC Server handler 5 on 58268) [    ] BlockStateChange BLOCK* processReport: from storage DS-92485a70-7902-45ec-8acd-ed1781891861 node DatanodeRegistration(127.0.0.1:45128, datanodeUuid=a96c74dd-3aae-4291-b30e-2dd30c46165c, infoPort=58437, infoSecurePort=0, ipcPort=55155, storageInfo=lv=-56;cid=testClusterID;nsid=1586924895;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3264851 INFO  (IPC Server handler 5 on 58268) [    ] BlockStateChange BLOCK* processReport: from storage DS-adc82f10-0310-4388-b03e-12c5f37db649 node DatanodeRegistration(127.0.0.1:45128, datanodeUuid=a96c74dd-3aae-4291-b30e-2dd30c46165c, infoPort=58437, infoSecurePort=0, ipcPort=55155, storageInfo=lv=-56;cid=testClusterID;nsid=1586924895;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3265234 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46818
   [junit4]   2> 3265332 INFO  (IPC Server handler 1 on 58268) [    ] BlockStateChange BLOCK* processReport: from storage DS-48d694c5-cddf-4d1b-afaa-5b3fbcb0dcf0 node DatanodeRegistration(127.0.0.1:54906, datanodeUuid=96d57b7e-b84b-4eb2-b51b-3313ccecfc0f, infoPort=39947, infoSecurePort=0, ipcPort=60838, storageInfo=lv=-56;cid=testClusterID;nsid=1586924895;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3265332 INFO  (IPC Server handler 1 on 58268) [    ] BlockStateChange BLOCK* processReport: from storage DS-28197cdc-eeae-40b2-b933-7aade54cd990 node DatanodeRegistration(127.0.0.1:54906, datanodeUuid=96d57b7e-b84b-4eb2-b51b-3313ccecfc0f, infoPort=39947, infoSecurePort=0, ipcPort=60838, storageInfo=lv=-56;cid=testClusterID;nsid=1586924895;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3265452 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3265452 INFO  (Thread-77741) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3265452 INFO  (Thread-77741) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3265455 ERROR (Thread-77741) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3265552 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.ZkTestServer start zk server on port:54189
   [junit4]   2> 3265568 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3265571 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3265573 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3265574 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3265578 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3265579 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3265581 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3265583 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3265585 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3265586 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3265588 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3265721 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/control-001/cores/collection1
   [junit4]   2> 3265723 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3265725 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@76009619{/_aan/n,null,AVAILABLE}
   [junit4]   2> 3265725 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@27812d67{SSL,[ssl, http/1.1]}{127.0.0.1:33415}
   [junit4]   2> 3265725 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.Server Started @3268906ms
   [junit4]   2> 3265725 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:58268/hdfs__localhost_58268__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001_tempDir-002_control_data, hostContext=/_aan/n, hostPort=33415, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/control-001/cores}
   [junit4]   2> 3265727 ERROR (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3265728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3265728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3265728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3265728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-17T02:14:27.967Z
   [junit4]   2> 3265731 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3265731 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/control-001/solr.xml
   [junit4]   2> 3265737 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3265741 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54189/solr
   [junit4]   2> 3265764 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3265764 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33415__aan%2Fn
   [junit4]   2> 3265766 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.Overseer Overseer (id=97975485575528453-127.0.0.1:33415__aan%2Fn-n_0000000000) starting
   [junit4]   2> 3265781 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33415__aan%2Fn
   [junit4]   2> 3265784 INFO  (zkCallback-2546-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3265869 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3265875 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3265875 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3265877 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/control-001/cores
   [junit4]   2> 3265877 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3265878 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3265881 INFO  (OverseerStateUpdate-97975485575528453-127.0.0.1:33415__aan%2Fn-n_0000000000) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3266901 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3266917 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3267042 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3267057 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3267058 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.core_node1' (registry 'solr.core.control_collection.shard1.core_node1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3267058 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58268/solr_hdfs_home
   [junit4]   2> 3267058 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3267058 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3267058 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/control-001/cores/collection1], dataDir=[hdfs://localhost:58268/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3267059 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58268/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 3267069 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3267069 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3267069 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3267078 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3267079 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58268/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3267100 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58268/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3267108 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3267108 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3267108 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3267115 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3267116 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20417212861826234]
   [junit4]   2> 3267135 INFO  (IPC Server handler 3 on 58268) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45128 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48d694c5-cddf-4d1b-afaa-5b3fbcb0dcf0:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-92485a70-7902-45ec-8acd-ed1781891861:NORMAL:127.0.0.1:45128|RBW]]} size 0
   [junit4]   2> 3267137 INFO  (IPC Server handler 6 on 58268) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48d694c5-cddf-4d1b-afaa-5b3fbcb0dcf0:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-92485a70-7902-45ec-8acd-ed1781891861:NORMAL:127.0.0.1:45128|RBW]]} size 0
   [junit4]   2> 3267143 WARN  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3267200 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3267200 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3267200 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3267214 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3267214 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3267217 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@23c9165a
   [junit4]   2> 3267223 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@32edaa4e[collection1] main]
   [junit4]   2> 3267224 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3267225 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3267225 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3267227 INFO  (searcherExecutor-7367-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@32edaa4e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3267227 INFO  (coreLoadExecutor-7366-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567607771067580416
   [junit4]   2> 3267236 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3267236 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3267236 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33415/_aan/n/collection1/
   [junit4]   2> 3267237 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3267237 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:33415/_aan/n/collection1/ has no replicas
   [junit4]   2> 3267237 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3267241 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33415/_aan/n/collection1/ shard1
   [junit4]   2> 3267392 INFO  (coreZkRegister-7359-thread-1-processing-n:127.0.0.1:33415__aan%2Fn x:collection1 c:control_collection) [n:127.0.0.1:33415__aan%2Fn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3267404 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3267405 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54189/solr ready
   [junit4]   2> 3267405 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3267405 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 3267559 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/shard-1-001/cores/collection1
   [junit4]   2> 3267560 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/shard-1-001
   [junit4]   2> 3267562 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3267564 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@52c604b9{/_aan/n,null,AVAILABLE}
   [junit4]   2> 3267565 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3e896c15{SSL,[ssl, http/1.1]}{127.0.0.1:50195}
   [junit4]   2> 3267567 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.e.j.s.Server Started @3270746ms
   [junit4]   2> 3267567 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:58268/hdfs__localhost_58268__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/_aan/n, hostPort=50195, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/shard-1-001/cores}
   [junit4]   2> 3267567 ERROR (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3267567 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 3267568 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3267568 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3267568 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-17T02:14:29.807Z
   [junit4]   2> 3267571 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3267571 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/shard-1-001/solr.xml
   [junit4]   2> 3267577 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3267580 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54189/solr
   [junit4]   2> 3267591 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3267593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3267596 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50195__aan%2Fn
   [junit4]   2> 3267598 INFO  (zkCallback-2546-thread-1-processing-n:127.0.0.1:33415__aan%2Fn) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3267598 INFO  (zkCallback-2550-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3267600 INFO  (zkCallback-2556-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3267677 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3267684 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3267684 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3267687 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/shard-1-001/cores
   [junit4]   2> 3267687 INFO  (TEST-HdfsSyncSliceTest.test-seed#[4BCC2C9A43BDECB1]) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3267696 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3267698 INFO  (OverseerStateUpdate-97975485575528453-127.0.0.1:33415__aan%2Fn-n_0000000000) [n:127.0.0.1:33415__aan%2Fn    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3267801 INFO  (zkCallback-2556-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3268715 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3268733 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3268854 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3268868 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1, trusted=true
   [junit4]   2> 3268869 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.core_node1' (registry 'solr.core.collection1.shard1.core_node1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f6a3751
   [junit4]   2> 3268869 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58268/solr_hdfs_home
   [junit4]   2> 3268869 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3268869 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3268869 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:58268/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 3268871 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58268/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 3268880 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3268880 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3268880 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3268885 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3268886 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58268/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 3268905 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58268/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 3268913 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3268913 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3268913 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3268917 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3268917 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20417212861826234]
   [junit4]   2> 3268932 INFO  (IPC Server handler 7 on 58268) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45128 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-48d694c5-cddf-4d1b-afaa-5b3fbcb0dcf0:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-adc82f10-0310-4388-b03e-12c5f37db649:NORMAL:127.0.0.1:45128|RBW]]} size 0
   [junit4]   2> 3268933 INFO  (IPC Server handler 8 on 58268) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-adc82f10-0310-4388-b03e-12c5f37db649:NORMAL:127.0.0.1:45128|RBW], ReplicaUC[[DISK]DS-28197cdc-eeae-40b2-b933-7aade54cd990:NORMAL:127.0.0.1:54906|FINALIZED]]} size 0
   [junit4]   2> 3268938 WARN  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3268995 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3268995 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3268995 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3269009 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3269009 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3269011 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6ae71db7
   [junit4]   2> 3269015 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6efa7ce0[collection1] main]
   [junit4]   2> 3269016 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3269017 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3269017 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3269019 INFO  (searcherExecutor-7378-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6efa7ce0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3269019 INFO  (coreLoadExecutor-7377-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1567607772946628608
   [junit4]   2> 3269026 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3269026 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3269027 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:50195/_aan/n/collection1/
   [junit4]   2> 3269027 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3269027 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:50195/_aan/n/collection1/ has no replicas
   [junit4]   2> 3269027 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3269031 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:50195/_aan/n/collection1/ shard1
   [junit4]   2> 3269136 INFO  (zkCallback-2556-thread-1-processing-n:127.0.0.1:50195__aan%2Fn) [n:127.0.0.1:50195__aan%2Fn    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3269182 INFO  (coreZkRegister-7372-thread-1-processing-n:127.0.0.1:50195__aan%2Fn x:collection1 c:collection1) [n:127.0.0.1:50195__aan%2Fn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery ne

[...truncated too long message...]

rceBuilder.java:79)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.teardownClass(HdfsSyncSliceTest.java:46)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> 3323617 INFO  (SUITE-HdfsSyncSliceTest-seed#[4BCC2C9A43BDECB1]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsSyncSliceTest_4BCC2C9A43BDECB1-001
   [junit4]   2> May 17, 2017 2:15:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=703, maxMBSortInHeap=7.8662547450505045, sim=RandomSimilarity(queryNorm=false): {}, locale=es-SV, timezone=America/Danmarkshavn
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=347220968,total=526385152
   [junit4]   2> NOTE: All tests run in this JVM: [TestBulkSchemaConcurrent, TestRangeQuery, TestCloudInspectUtil, AnalysisAfterCoreReloadTest, TestConfigSetImmutable, TestFieldCache, TestSolrCloudWithKerberosAlt, TestCorePropertiesReload, SimpleMLTQParserTest, QueryEqualityTest, TestTolerantUpdateProcessorRandomCloud, MultiTermTest, TestDefaultStatsCache, TestUseDocValuesAsStored, SolrRequestParserTest, TestCloudNestedDocsSort, NumericFieldsTest, TestSolrCloudSnapshots, TestPivotHelperCode, OverseerRolesTest, TestJettySolrRunner, OverriddenZkACLAndCredentialsProvidersTest, ZkStateWriterTest, ShufflingReplicaListTransformerTest, HighlighterMaxOffsetTest, RegexBoostProcessorTest, TestPathTrie, TestJmxIntegration, TestOverriddenPrefixQueryForCustomFieldType, TestFaceting, TestJoin, TestPKIAuthenticationPlugin, TestAddFieldRealTimeGet, CopyFieldTest, TestChildDocTransformer, TestDocumentBuilder, SparseHLLTest, SolrMetricsIntegrationTest, TestUniqueKeyFieldResource, CoreAdminCreateDiscoverTest, DistributedQueryElevationComponentTest, ClassificationUpdateProcessorIntegrationTest, TestAnalyzedSuggestions, TestXmlQParser, TestRTimerTree, ConfigSetsAPITest, CollectionsAPIAsyncDistributedZkTest, TestSolr4Spatial2, TestCSVLoader, TestReplicationHandlerBackup, TestBadConfig, DateRangeFieldTest, TestSolrDeletionPolicy2, TestFunctionQuery, TestFoldingMultitermQuery, TestImplicitCoreProperties, TestHdfsUpdateLog, NotRequiredUniqueKeyTest, TestIntervalFaceting, SolrCoreTest, PeerSyncTest, HttpSolrCallGetCoreTest, TestPointFields, TestBlobHandler, SpellCheckCollatorWithCollapseTest, TestRemoteStreaming, TestStressLucene, DeleteNodeTest, TestSimpleTextCodec, TestFilteredDocIdSet, SolrShardReporterTest, SyncSliceTest, TestValueSourceCache, PingRequestHandlerTest, TestSolr4Spatial, TestReRankQParserPlugin, TestManagedSchema, XsltUpdateRequestHandlerTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsDirectoryTest, TestRawResponseWriter, DocumentAnalysisRequestHandlerTest, TestScoreJoinQPScore, TestRandomDVFaceting, TestNamedUpdateProcessors, TestCoreAdminApis, TestSuggestSpellingConverter, InfoHandlerTest, TestLFUCache, CurrencyFieldOpenExchangeTest, TestDistribDocBasedVersion, TestSortByMinMaxFunction, BasicZkTest, MissingSegmentRecoveryTest, BasicAuthStandaloneTest, BlockJoinFacetRandomTest, TestStressInPlaceUpdates, TestLazyCores, SolrCoreMetricManagerTest, TimeZoneUtilsTest, TestLeaderInitiatedRecoveryThread, TestCodecSupport, BigEndianAscendingWordSerializerTest, DeleteStatusTest, FullHLLTest, TestCopyFieldCollectionResource, UninvertDocValuesMergePolicyTest, TestFastWriter, TestSubQueryTransformerDistrib, DateMathParserTest, TestOnReconnectListenerSupport, LeaderInitiatedRecoveryOnCommitTest, PolyFieldTest, ChaosMonkeySafeLeaderTest, LeaderElectionIntegrationTest, TestReplicationHandler, TestHighlightDedupGrouping, TestTolerantSearch, ConnectionReuseTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest, CdcrVersionReplicationTest, CollectionTooManyReplicasTest, DeleteReplicaTest, DistribCursorPagingTest, OverseerModifyCollectionTest, OverseerStatusTest, OverseerTaskQueueTest, ReplaceNodeTest, SSLMigrationTest, ShardSplitTest, TestRebalanceLeaders, TestSSLRandomization, TestSegmentSorting, TestSolrCloudWithSecureImpersonation, TestStressCloudBlindAtomicUpdates, HdfsBasicDistributedZkTest, HdfsSyncSliceTest]
   [junit4] Completed [547/709 (1!)] on J0 in 69.71s, 1 test, 1 failure <<< FAILURES!

[...truncated 43921 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1303 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1303/

1 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1495060768577,generation=2,filelist=[_bt.cfe, _bt.cfs, _bt.si, _c4.cfe, _c4.cfs, _c4.si, _c5.cfe, _c5.cfs, _c5.si, _c8.cfe, _c8.cfs, _c8.si, _ca.cfe, _ca.cfs, _ca.si, _cb.cfe, _cb.cfs, _cb.si, _cc.cfe, _cc.cfs, _cc.si, _cd.cfe, _cd.cfs, _cd.si, _ce.cfe, _ce.cfs, _ce.si, _cf.cfe, _cf.cfs, _cf.si, _cg.cfe, _cg.cfs, _cg.si, _ci.cfe, _ci.cfs, _ci.si, _cj.cfe, _cj.cfs, _cj.si, _ck.cfe, _ck.cfs, _ck.si, _cl.cfe, _cl.cfs, _cl.si, _cn.cfe, _cn.cfs, _cn.si, _co.fdt, _co.fdx, _co.fnm, _co.nvd, _co.nvm, _co.si, _co_FST50_0.doc, _co_FST50_0.tfp, _cq.cfe, _cq.cfs, _cq.si, _cr.cfe, _cr.cfs, _cr.si, _cs.cfe, _cs.cfs, _cs.si, _ct.cfe, _ct.cfs, _ct.si, _cu.cfe, _cu.cfs, _cu.si, _cv.cfe, _cv.cfs, _cv.si, _cx.cfe, _cx.cfs, _cx.si, _cz.cfe, _cz.cfs, _cz.si, _d0.cfe, _d0.cfs, _d0.si, _d1.cfe, _d1.cfs, _d1.si, _d2.cfe, _d2.cfs, _d2.si, segments_2]}]> but was:<[{indexVersion=1495060768577,generation=2,filelist=[_bt.cfe, _bt.cfs, _bt.si, _c4.cfe, _c4.cfs, _c4.si, _c5.cfe, _c5.cfs, _c5.si, _c8.cfe, _c8.cfs, _c8.si, _ca.cfe, _ca.cfs, _ca.si, _cb.cfe, _cb.cfs, _cb.si, _cc.cfe, _cc.cfs, _cc.si, _cd.cfe, _cd.cfs, _cd.si, _ce.cfe, _ce.cfs, _ce.si, _cf.cfe, _cf.cfs, _cf.si, _cg.cfe, _cg.cfs, _cg.si, _ci.cfe, _ci.cfs, _ci.si, _cj.cfe, _cj.cfs, _cj.si, _ck.cfe, _ck.cfs, _ck.si, _cl.cfe, _cl.cfs, _cl.si, _cn.cfe, _cn.cfs, _cn.si, _co.fdt, _co.fdx, _co.fnm, _co.nvd, _co.nvm, _co.si, _co_FST50_0.doc, _co_FST50_0.tfp, _cq.cfe, _cq.cfs, _cq.si, _cr.cfe, _cr.cfs, _cr.si, _cs.cfe, _cs.cfs, _cs.si, _ct.cfe, _ct.cfs, _ct.si, _cu.cfe, _cu.cfs, _cu.si, _cv.cfe, _cv.cfs, _cv.si, _cx.cfe, _cx.cfs, _cx.si, _cz.cfe, _cz.cfs, _cz.si, _d0.cfe, _d0.cfs, _d0.si, _d1.cfe, _d1.cfs, _d1.si, _d2.cfe, _d2.cfs, _d2.si, segments_2]}, {indexVersion=1495060768577,generation=3,filelist=[_cy.fdt, _cy.fdx, _cy.fnm, _cy.nvd, _cy.nvm, _cy.si, _cy_FST50_0.doc, _cy_FST50_0.tfp, _d4.cfe, _d4.cfs, _d4.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1495060768577,generation=2,filelist=[_bt.cfe, _bt.cfs, _bt.si, _c4.cfe, _c4.cfs, _c4.si, _c5.cfe, _c5.cfs, _c5.si, _c8.cfe, _c8.cfs, _c8.si, _ca.cfe, _ca.cfs, _ca.si, _cb.cfe, _cb.cfs, _cb.si, _cc.cfe, _cc.cfs, _cc.si, _cd.cfe, _cd.cfs, _cd.si, _ce.cfe, _ce.cfs, _ce.si, _cf.cfe, _cf.cfs, _cf.si, _cg.cfe, _cg.cfs, _cg.si, _ci.cfe, _ci.cfs, _ci.si, _cj.cfe, _cj.cfs, _cj.si, _ck.cfe, _ck.cfs, _ck.si, _cl.cfe, _cl.cfs, _cl.si, _cn.cfe, _cn.cfs, _cn.si, _co.fdt, _co.fdx, _co.fnm, _co.nvd, _co.nvm, _co.si, _co_FST50_0.doc, _co_FST50_0.tfp, _cq.cfe, _cq.cfs, _cq.si, _cr.cfe, _cr.cfs, _cr.si, _cs.cfe, _cs.cfs, _cs.si, _ct.cfe, _ct.cfs, _ct.si, _cu.cfe, _cu.cfs, _cu.si, _cv.cfe, _cv.cfs, _cv.si, _cx.cfe, _cx.cfs, _cx.si, _cz.cfe, _cz.cfs, _cz.si, _d0.cfe, _d0.cfs, _d0.si, _d1.cfe, _d1.cfs, _d1.si, _d2.cfe, _d2.cfs, _d2.si, segments_2]}]> but was:<[{indexVersion=1495060768577,generation=2,filelist=[_bt.cfe, _bt.cfs, _bt.si, _c4.cfe, _c4.cfs, _c4.si, _c5.cfe, _c5.cfs, _c5.si, _c8.cfe, _c8.cfs, _c8.si, _ca.cfe, _ca.cfs, _ca.si, _cb.cfe, _cb.cfs, _cb.si, _cc.cfe, _cc.cfs, _cc.si, _cd.cfe, _cd.cfs, _cd.si, _ce.cfe, _ce.cfs, _ce.si, _cf.cfe, _cf.cfs, _cf.si, _cg.cfe, _cg.cfs, _cg.si, _ci.cfe, _ci.cfs, _ci.si, _cj.cfe, _cj.cfs, _cj.si, _ck.cfe, _ck.cfs, _ck.si, _cl.cfe, _cl.cfs, _cl.si, _cn.cfe, _cn.cfs, _cn.si, _co.fdt, _co.fdx, _co.fnm, _co.nvd, _co.nvm, _co.si, _co_FST50_0.doc, _co_FST50_0.tfp, _cq.cfe, _cq.cfs, _cq.si, _cr.cfe, _cr.cfs, _cr.si, _cs.cfe, _cs.cfs, _cs.si, _ct.cfe, _ct.cfs, _ct.si, _cu.cfe, _cu.cfs, _cu.si, _cv.cfe, _cv.cfs, _cv.si, _cx.cfe, _cx.cfs, _cx.si, _cz.cfe, _cz.cfs, _cz.si, _d0.cfe, _d0.cfs, _d0.si, _d1.cfe, _d1.cfs, _d1.si, _d2.cfe, _d2.cfs, _d2.si, segments_2]}, {indexVersion=1495060768577,generation=3,filelist=[_cy.fdt, _cy.fdx, _cy.fnm, _cy.nvd, _cy.nvm, _cy.si, _cy_FST50_0.doc, _cy_FST50_0.tfp, _d4.cfe, _d4.cfs, _d4.si, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([2B89B0A0FA64B39B:E5EAB908A2CBD98]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1279)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12780 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/init-core-data-001
   [junit4]   2> 2175822 WARN  (SUITE-TestReplicationHandler-seed#[2B89B0A0FA64B39B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=59 numCloses=59
   [junit4]   2> 2175822 INFO  (SUITE-TestReplicationHandler-seed#[2B89B0A0FA64B39B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 2175824 INFO  (SUITE-TestReplicationHandler-seed#[2B89B0A0FA64B39B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 2175826 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestIndexAndConfigReplication
   [junit4]   2> 2175826 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/collection1
   [junit4]   2> 2175835 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2175939 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@52198d29{/solr,null,AVAILABLE}
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4090b419{HTTP/1.1,[http/1.1]}{127.0.0.1:39886}
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.Server Started @2179051ms
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=39886}
   [junit4]   2> 2175940 ERROR (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2175940 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-17T22:37:44.648Z
   [junit4]   2> 2175951 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/solr.xml
   [junit4]   2> 2175957 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2176007 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2176013 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2176013 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2176016 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/.
   [junit4]   2> 2176016 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2176028 WARN  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 2176032 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2176062 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2176065 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2176065 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 2176065 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2176065 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2176066 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 2176067 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=4, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2176121 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2176121 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2176122 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@49312a76
   [junit4]   2> 2176122 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2dd28556[collection1] main]
   [junit4]   2> 2176123 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-001/collection1/conf
   [junit4]   2> 2176124 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 2176124 INFO  (coreLoadExecutor-4268-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2176125 INFO  (searcherExecutor-4269-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2dd28556[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2176626 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/collection1
   [junit4]   2> 2176629 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2176648 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@46b26e82{/solr,null,AVAILABLE}
   [junit4]   2> 2176648 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7307dc19{HTTP/1.1,[http/1.1]}{127.0.0.1:38487}
   [junit4]   2> 2176649 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.Server Started @2179760ms
   [junit4]   2> 2176649 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=38487}
   [junit4]   2> 2176649 ERROR (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2176749 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2176749 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 2176749 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2176749 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-05-17T22:37:45.457Z
   [junit4]   2> 2176750 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/solr.xml
   [junit4]   2> 2176758 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2176882 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2177052 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2177052 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2177055 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/.
   [junit4]   2> 2177067 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2177084 WARN  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 2177097 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2177121 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2177271 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2177271 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 2177271 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@f1a9665
   [junit4]   2> 2177271 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2177271 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 2177303 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=4, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2177411 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2177411 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2177413 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@22f01320
   [junit4]   2> 2177413 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@58f4380c[collection1] main]
   [junit4]   2> 2177414 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001/solr-instance-002/collection1/conf
   [junit4]   2> 2177419 INFO  (coreLoadExecutor-4278-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 2177421 INFO  (searcherExecutor-4279-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@58f4380c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2177616 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2177618 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 1
   [junit4]   2> 2177619 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 2177636 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 2177655 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 2177656 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 2177656 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 2177657 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 2177657 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 2177657 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 2177658 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 2177658 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 2177659 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 2177667 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 2177667 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 2177669 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 2177671 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 2177674 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 2177676 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 2177677 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 2177677 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 2177678 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 2177678 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 2177678 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 2177679 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 2177680 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 2177680 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 2177680 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 2177681 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 2177681 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 2177682 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 2177682 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 2177683 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 2177683 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 2177684 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 2177684 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 2177685 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 2177685 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 2177686 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 2177686 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 2177686 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 2177687 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 2177687 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 2177688 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 2177688 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 2177689 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 2177689 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 2177690 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 2177690 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 2177691 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 2177692 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 2177692 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 2177693 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 2177694 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 2177694 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 2177695 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 2177696 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 2177697 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 2177697 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 2177698 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 2177698 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 2177699 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 2177702 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 2177703 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 2177710 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 2177711 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 2177712 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 2177712 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 2177713 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 2177715 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 2177715 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 2177716 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 2177716 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 2177717 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 2177718 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 2177719 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 2177720 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 2177720 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 2177721 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 2177721 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 2177722 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 2177723 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 2177724 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 2177724 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 2177725 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 2177725 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 2177726 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 2177726 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 2177727 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 2177728 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 2177729 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 2177730 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 2177730 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 2177731 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 2177732 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 2177732 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 2177733 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 2177734 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 2177734 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 2177735 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 2177735 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 2177742 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 2177744 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 2177744 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 2177745 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 2177746 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 2177747 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 2177748 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 2177749 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 2177750 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 2177750 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 2177751 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 2177752 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 2177767 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 2177769 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 2177770 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 2177772 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 2177773 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 2177775 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 2177776 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 2177778 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 2177779 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 2177780 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 2177781 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 2177782 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 2177784 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 2177785 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 2177786 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 2177788 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 2177789 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 2177790 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 2177792 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 2177794 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 2177794 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 2177795 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 2177798 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 0
   [junit4]   2> 2177798 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 0
   [junit4]   2> 2177799 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 2177801 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 2177801 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 2177802 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 2177803 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 2177803 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 2177804 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 2177806 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 0
   [junit4]   2> 2177808 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 2177808 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 2177809 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 2177810 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 2177814 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 2177814 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 2177815 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
   [junit4]   2> 2177815 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 0
   [junit4]   2> 2177816 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
   [junit4]   2> 2177817 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 0
   [junit4]   2> 2177818 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
   [junit4]   2> 2177819 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
   [junit4]   2> 2177819 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
   [junit4]   2> 2177820 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
   [junit4]   2> 2177821 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
   [junit4]   2> 2177822 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 0
   [junit4]   2> 2177823 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
   [junit4]   2> 2177823 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
   [junit4]   2> 2177824 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
   [junit4]   2> 2177825 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
   [junit4]   2> 2177827 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
   [junit4]   2> 2177828 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
   [junit4]   2> 2177828 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 0
   [junit4]   2> 2177828 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[167]} 0 0
   [junit4]   2> 2177829 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[168]} 0 0
   [junit4]   2> 2177829 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[169]} 0 0
   [junit4]   2> 2177830 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[170]} 0 0
   [junit4]   2> 2177830 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[171]} 0 0
   [junit4]   2> 2177830 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[172]} 0 0
   [junit4]   2> 2177831 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[173]} 0 0
   [junit4]   2> 2177831 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[174]} 0 0
   [junit4]   2> 2177832 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[175]} 0 0
   [junit4]   2> 2177832 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[176]} 0 0
   [junit4]   2> 2177832 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[177]} 0 0
   [junit4]   2> 2177833 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[178]} 0 0
   [junit4]   2> 2177833 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[179]} 0 0
   [junit4]   2> 2177834 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[180]} 0 0
   [junit4]   2> 2177834 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[181]} 0 0
   [junit4]   2> 2177834 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[182]} 0 0
   [junit4]   2> 2177835 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[183]} 0 0
   [junit4]   2> 2177835 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[184]} 0 0
   [junit4]   2> 2177836 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[185]} 0 0
   [junit4]   2> 2177836 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[186]} 0 0
   [junit4]   2> 2177836 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[187]} 0 0
   [junit4]   2> 2177837 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[188]} 0 0
   [junit4]   2> 2177837 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[189]} 0 0
   [junit4]   2> 2177837 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[190]} 0 0
   [junit4]   2> 2177838 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[191]} 0 0
   [junit4]   2> 2177838 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[192]} 0 0
   [junit4]   2> 2177839 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[193]} 0 0
   [junit4]   2> 2177839 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[194]} 0 0
   [junit4]   2> 2177839 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[195]} 0 0
   [junit4]   2> 2177840 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[196]} 0 0
   [junit4]   2> 2177840 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[197]} 0 0
   [junit4]   2> 2177840 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[198]} 0 0
   [junit4]   2> 2177841 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[199]} 0 0
   [junit4]   2> 2177841 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[200]} 0 0
   [junit4]   2> 2177842 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[201]} 0 0
   [junit4]   2> 2177842 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[202]} 0 0
   [junit4]   2> 2177842 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[203]} 0 0
   [junit4]   2> 2177843 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[204]} 0 0
   [junit4]   2> 2177843 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[205]} 0 0
   [junit4]   2> 2177843 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[206]} 0 0
   [junit4]   2> 2177844 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[207]} 0 0
   [junit4]   2> 2177844 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[208]} 0 0
   [junit4]   2> 2177845 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[209]} 0 0
   [junit4]   2> 2177845 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[210]} 0 0
   [junit4]   2> 2177845 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[211]} 0 0
   [junit4]   2> 2177846 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[212]} 0 0
   [junit4]   2> 2177846 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[213]} 0 0
   [junit4]   2> 2177847 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[214]} 0 0
   [junit4]   2> 2177847 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[215]} 0 0
   [junit4]   2> 2177847 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[216]} 0 0
   [junit4]   2> 2177850 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[217]} 0 0
   [junit4]   2> 2177851 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[218]} 0 0
   [junit4]   2> 2177851 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[219]} 0 0
   [junit4]   2> 2177852 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[220]} 0 0
   [junit4]   2> 2177852 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[221]} 0 0
   [junit4]   2> 2177853 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[222]} 0 0
   [junit4]   2> 2177853 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[223]} 0 0
   [junit4]   2> 2177853 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[224]} 0 0
   [junit4]   2> 2177854 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[225]} 0 0
   [junit4]   2> 2177854 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[226]} 0 0
   [junit4]   2> 2177855 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[227]} 0 0
   [junit4]   2> 2177855 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[228]} 0 0
   [junit4]   2> 2177855 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[229]} 0 0
   [junit4]   2> 2177856 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[230]} 0 0
   [junit4]   2> 2177856 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[231]} 0 0
   [junit4]   2> 2177857 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[232]} 0 0
   [junit4]   2> 2177857 INFO  (qtp913025236-160809) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2177857 INFO  (indexFetcher-4283-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 2177858 INFO  (indexFetcher-4283-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 2177858 INFO  (indexFetcher-4283-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2177858 INFO  (indexFetcher-4283-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2177858 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[233]} 0 0
   [junit4]   2> 2177858 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[234]} 0 0
   [junit4]   2> 2177859 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[235]} 0 0
   [junit4]   2> 2177859 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[236]} 0 0
   [junit4]   2> 2177859 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[237]} 0 0
   [junit4]   2> 2177860 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[238]} 0 0
   [junit4]   2> 2177860 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[239]} 0 0
   [junit4]   2> 2177861 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[240]} 0 0
   [junit4]   2> 2177862 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[241]} 0 0
   [junit4]   2> 2177881 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[242]} 0 0
   [junit4]   2> 2177882 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[243]} 0 0
   [junit4]   2> 2177883 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[244]} 0 0
   [junit4]   2> 2177886 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[245]} 0 0
   [junit4]   2> 2177888 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[246]} 0 0
   [junit4]   2> 2177891 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[247]} 0 0
   [junit4]   2> 2177891 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[248]} 0 0
   [junit4]   2> 2177892 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[249]} 0 0
   [junit4]   2> 2177892 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[250]} 0 0
   [junit4]   2> 2177893 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[251]} 0 0
   [junit4]   2> 2177893 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[252]} 0 0
   [junit4]   2> 2177893 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[253]} 0 0
   [junit4]   2> 2177894 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[254]} 0 0
   [junit4]   2> 2177899 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[255]} 0 0
   [junit4]   2> 2177900 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[256]} 0 0
   [junit4]   2> 2177907 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[257]} 0 0
   [junit4]   2> 2177908 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[258]} 0 0
   [junit4]   2> 2177908 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[259]} 0 0
   [junit4]   2> 2177908 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[260]} 0 0
   [junit4]   2> 2177909 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[261]} 0 0
   [junit4]   2> 2177909 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[262]} 0 0
   [junit4]   2> 2177912 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[263]} 0 0
   [junit4]   2> 2177912 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[264]} 0 0
   [junit4]   2> 2177919 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[265]} 0 0
   [junit4]   2> 2177920 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[266]} 0 0
   [junit4]   2> 2177920 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[267]} 0 0
   [junit4]   2> 2177921 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[268]} 0 0
   [junit4]   2> 2177921 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[269]} 0 0
   [junit4]   2> 2177922 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[270]} 0 0
   [junit4]   2> 2177922 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[271]} 0 0
   [junit4]   2> 2177922 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[272]} 0 0
   [junit4]   2> 2177923 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[273]} 0 0
   [junit4]   2> 2177923 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[274]} 0 0
   [junit4]   2> 2177924 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[275]} 0 0
   [junit4]   2> 2177929 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[276]} 0 0
   [junit4]   2> 2177929 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[277]} 0 0
   [junit4]   2> 2177930 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[278]} 0 0
   [junit4]   2> 2177930 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[279]} 0 0
   [junit4]   2> 2177930 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[280]} 0 0
   [junit4]   2> 2177931 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[281]} 0 0
   [junit4]   2> 2177931 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[282]} 0 0
   [junit4]   2> 2177932 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[283]} 0 0
   [junit4]   2> 2177932 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[284]} 0 0
   [junit4]   2> 2177949 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[285]} 0 0
   [junit4]   2> 2177950 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[286]} 0 0
   [junit4]   2> 2177951 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[287]} 0 0
   [junit4]   2> 2177955 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[288]} 0 0
   [junit4]   2> 2177956 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[289]} 0 0
   [junit4]   2> 2177956 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[290]} 0 0
   [junit4]   2> 2177957 INFO  (qtp913025236-160813) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[291]} 0 0
   [junit4]   2> 2177958 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[292]} 0 0
   [junit4]   2> 2177958 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[293]} 0 0
   [junit4]   2> 2177959 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[294]} 0 0
   [junit4]   2> 2177979 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[295]} 0 0
   [junit4]   2> 2177980 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[296]} 0 0
   [junit4]   2> 2177980 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[297]} 0 0
   [junit4]   2> 2177981 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[298]} 0 0
   [junit4]   2> 2177982 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[299]} 0 0
   [junit4]   2> 2177982 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[300]} 0 0
   [junit4]   2> 2177985 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[301]} 0 0
   [junit4]   2> 2177985 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[302]} 0 0
   [junit4]   2> 2177986 INFO  (qtp913025236-160812) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[303]} 0 0
   [junit4]   2> 2177986 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[304]} 0 0
   [junit4]   2> 2177987 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[305]} 0 0
   [junit4]   2> 2177987 INFO  (qtp913025236-160814) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[306]} 0 0
   [junit4]   2> 2177991 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[307]} 0 0
   [junit4]   2> 2177995 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[308]} 0 0
   [junit4]   2> 2177995 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[309]} 0 0
   [junit4]   2> 2177996 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[310]} 0 0
   [junit4]   2> 2177996 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[311]} 0 0
   [junit4]   2> 2177997 INFO  (qtp913025236-160815) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[312]} 0 0
   [junit4]   2> 2177997 INFO  (qtp913025236-160808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[313]} 0 0
   [junit4]   2> 2177998 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[314]} 0 0
   [junit4]   2> 2177998 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[315]} 0 0
   [junit4]   2> 2177999 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[316]} 0 0
   [junit4]   2> 2177999 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[317]} 0 0
   [junit4]   2> 2178000 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[318]} 0 0
   [junit4]   2> 2178000 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[319]} 0 0
   [junit4]   2> 2178001 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[320]} 0 0
   [junit4]   2> 2178001 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[321]} 0 0
   [junit4]   2> 2178002 INFO  (qtp913025236-160811) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFact

[...truncated too long message...]

r.java:139)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>
   [junit4]   2> 2482589 INFO  (qtp1020774189-162491) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={tlogFile=../dir/traversal&command=filecontent} status=403 QTime=0
   [junit4]   2> 2482590 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestIllegalFilePaths
   [junit4]   2> 2482599 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@57578da3{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2482600 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=358832593
   [junit4]   2> 2482600 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2482602 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2482604 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2482605 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2482605 INFO  (coreCloseExecutor-5204-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5de7987d
   [junit4]   2> 2482608 INFO  (coreCloseExecutor-5204-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1575458941
   [junit4]   2> 2482614 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2fe5301d{/solr,null,UNAVAILABLE}
   [junit4]   2> 2482614 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3699b601{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2482615 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=768734290
   [junit4]   2> 2482615 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2482615 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2482615 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2482615 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2482616 INFO  (coreCloseExecutor-5206-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6533fc0e
   [junit4]   2> 2482617 INFO  (coreCloseExecutor-5206-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1697905678
   [junit4]   2> 2482618 INFO  (TEST-TestReplicationHandler.doTestIllegalFilePaths-seed#[2B89B0A0FA64B39B]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4bcdf42b{/solr,null,UNAVAILABLE}
   [junit4]   2> 2482619 INFO  (SUITE-TestReplicationHandler-seed#[2B89B0A0FA64B39B]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_2B89B0A0FA64B39B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=FST50, id=FST50, newname=FST50}, docValues:{}, maxPointsInLeafNode=1795, maxMBSortInHeap=6.2499505776846505, sim=RandomSimilarity(queryNorm=true): {}, locale=es-ES, timezone=Australia/Melbourne
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=171815232,total=500170752
   [junit4]   2> NOTE: All tests run in this JVM: [CircularListTest, ActionThrottleTest, PropertiesRequestHandlerTest, DeleteShardTest, TestSubQueryTransformerCrossCore, CdcrVersionReplicationTest, StatsReloadRaceTest, SharedFSAutoReplicaFailoverUtilsTest, EnumFieldTest, TestIBSimilarityFactory, TestCollationField, SolrTestCaseJ4Test, DistributedFacetPivotLongTailTest, FieldMutatingUpdateProcessorTest, TestHashQParserPlugin, TestDynamicFieldResource, HdfsCollectionsAPIDistributedZkTest, TestTrieFacet, DirectUpdateHandlerTest, TestConfig, HLLUtilTest, TestCloudDeleteByQuery, SuggesterTest, BinaryUpdateRequestHandlerTest, AutoCommitTest, TestBlendedInfixSuggestions, CdcrReplicationDistributedZkTest, TestBinaryResponseWriter, TestReversedWildcardFilterFactory, TestHighFrequencyDictionaryFactory, XmlUpdateRequestHandlerTest, OutputWriterTest, TestLuceneMatchVersion, MetricsConfigTest, TestBinaryField, HdfsBasicDistributedZkTest, TestManagedResourceStorage, LeaderElectionTest, StatelessScriptUpdateProcessorFactoryTest, HdfsThreadLeakTest, TestEmbeddedSolrServerSchemaAPI, TestFieldCacheWithThreads, ZkStateReaderTest, DateFieldTest, TestLeaderElectionZkExpiry, FieldAnalysisRequestHandlerTest, TestManagedSchemaThreadSafety, TestFieldTypeCollectionResource, SuggesterWFSTTest, TestSort, CurrencyFieldXmlFileTest, TestDocSet, TestSolrCloudWithDelegationTokens, TestExtendedDismaxParser, SuggesterTSTTest, TestSegmentSorting, ParsingFieldUpdateProcessorsTest, OverseerTaskQueueTest, MultiThreadedOCPTest, TestClassNameShortening, TestTolerantUpdateProcessorCloud, ChaosMonkeyNothingIsSafeTest, DebugComponentTest, TestFastOutputStream, TestReload, TestRandomRequestDistribution, TestExactSharedStatsCache, BigEndianAscendingWordDeserializerTest, TermVectorComponentDistributedTest, TestPushWriter, OverseerTest, TriLevelCompositeIdRoutingTest, TestExpandComponent, TestSolrCloudWithHadoopAuthPlugin, RequestLoggingTest, TestSchemaResource, NoCacheHeaderTest, PrimitiveFieldTypeTest, TestWriterPerf, TestTestInjection, TestHdfsBackupRestoreCore, HardAutoCommitTest, TestIndexSearcher, CloudExitableDirectoryReaderTest, UnloadDistributedZkTest, RuleEngineTest, HdfsWriteToMultipleCollectionsTest, LukeRequestHandlerTest, LargeFieldTest, CreateCollectionCleanupTest, OpenExchangeRatesOrgProviderTest, TestRequestForwarding, TestOrdValues, TestGeoJSONResponseWriter, DeleteLastCustomShardedReplicaTest, PluginInfoTest, BadIndexSchemaTest, BufferStoreTest, SuggestComponentContextFilterQueryTest, TestLockTree, SolrCoreCheckLockOnStartupTest, UUIDUpdateProcessorFallbackTest, SortSpecParsingTest, CloudMLTQParserTest, ClassificationUpdateProcessorFactoryTest, RequestHandlersTest, TestHalfAndHalfDocValues, TestGroupingSearch, BasicAuthIntegrationTest, TestUtils, TestComplexPhraseLeadingWildcard, TestScoreJoinQPNoScore, DistribDocExpirationUpdateProcessorTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, TestReplicationHandler]
   [junit4] Completed [391/709 (1!)] on J2 in 306.81s, 16 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 44420 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|